[GLUTEN-12465][CORE] Support optional stage InputStats plumbing in scan/input-iterator transformers

[taiyang-li]

Background

One of a small series of common-code changes to introduce a new backend — Bolt (ByteDance’s unified lakehouse analytics acceleration engine) — into the Gluten community. The series minimizes the delta to Gluten common code so Bolt can plug in cleanly, while leaving Velox and ClickHouse backends unaffected. It is part of plan in #12456, close #12465

This PR

Introduce an opt-in framework to pass stage input statistics (scan / shuffle / broadcast) to the native engine as estimated row-size hints.

• Add InputStats / InputStatsKind / ShuffleStageWrapper and ApplyStageInputStatsRule.
• Add RelNode.childNode() with implementations across all rel nodes; carry rowSize / inputStats on ReadRelNode and InputIteratorRelNode; expose PlanNode.getRelNodes() and AdvancedExtensionNode optimization/enhancement getters.
• Add BasicScanExecTransformer.getInputStats hook and optional inputStats field on FileSourceScanExecTransformer; thread wsContext through genFirstStageIterator and the whole-stage RDDs; inject shuffle/broadcast stats in InputIteratorTransformer.
• Gate everything behind spark.gluten.sql.enablePassStageInputStats (default false), so Velox / ClickHouse behavior is unchanged.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Pull request overview

This PR adds an opt-in plumbing path (gated by spark.gluten.sql.enablePassStageInputStats, default false) to propagate scan/shuffle/broadcast stage input statistics through Gluten’s transformers into Substrait rel nodes, so native engines can consume an estimated input hint.

Changes:

• Introduces InputStats/InputStatsKind, ShuffleStageWrapper, and ApplyStageInputStatsRule to capture and apply stage statistics (including AQE shuffle read adjustments).
• Threads WholeStageTransformContext through first-stage iterators / whole-stage RDD compute paths so per-partition hints can be applied before native execution.
• Extends Substrait rel node traversal (RelNode.childNode()) and injects a rowSize=... optimization string into ReadRel advanced extensions.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ColumnarCollapseTransformStages.scala	Extracts stage runtime stats in InputIteratorTransformer and attaches optional InputStats to InputIteratorRelNode.
gluten-substrait/src/main/scala/org/apache/gluten/extension/ApplyStageInputStatsRule.scala	New rule to traverse rel nodes and apply per-partition estimated hints derived from stage stats.
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageZippedPartitionsRDD.scala	Applies stage input stats (when enabled) during final-stage RDD compute.
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala	Threads wsCtx and partitionLength into whole-stage RDD creation paths.
gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala	Applies stage input stats (when enabled) during first-stage RDD compute and passes wsContext into iterator API.
gluten-substrait/src/main/scala/org/apache/gluten/execution/FileSourceScanExecTransformer.scala	Adds optional inputStats field and exposes it via getInputStats.
gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala	Adds getInputStats hook and sets scan stats onto ReadRelNode when available.
gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala	Adds enablePassStageInputStats config entry and accessor.
gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/IteratorApi.scala	Extends iterator API signature to accept WholeStageTransformContext.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WriteRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowGroupLimitRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/TopNNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SortRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SetRelNode.java	Implements childNode() for rel traversal over multi-input rels.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/RelNode.java	Adds childNode() to the rel node interface.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ReadRelNode.java	Adds rowSize/inputStats fields and injects rowSize into advanced extensions.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ProjectRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/JoinRelNode.java	Implements childNode() returning both join children.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/InputIteratorRelNode.java	Adds rowSize/inputStats fields and injects rowSize into advanced extensions.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/GenerateRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FilterRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FetchRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ExpandRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/CrossRelNode.java	Implements childNode() returning both cross children.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/AggregateRelNode.java	Implements childNode() for rel traversal.
gluten-substrait/src/main/java/org/apache/gluten/substrait/plan/PlanNode.java	Exposes rel list via getRelNodes() to support traversal from the whole-stage root.
gluten-substrait/src/main/java/org/apache/gluten/substrait/extensions/AdvancedExtensionNode.java	Adds getters for optimization/enhancement Any payloads.
gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShuffleStageWrapper.scala	New wrapper and stats types to extract AQE stage runtime stats.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala	Updates iterator API signature to accept wsContext (currently unused in this backend).
backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHIteratorApi.scala	Updates iterator API signature to accept wsContext (currently unused in this backend).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[marin-ma]

@taiyang-li Thanks. Just add some minor comments. Can you please also address the comments from Copilot?

[github-actions]

Run Gluten Clickhouse CI on x86

[jackylee-ch]

@taiyang-li Hello, do we have any issues or design documents outlining the works required to introduce the BOLT engine? Also, it would be best to discuss and confirm the plan before starting the work.

A more suitable approach would be to outline an issue and the corresponding design documents based on extending the Gluten backends interface to support additional backends. We can then discuss this and proceed with the discussion on Bolt only after the initial integration is complete.

[taiyang-li]

@jackylee-ch Thanks for pointing this out.

We have created a tracking issue here: #12456, and I also updated the PR description in #12454 accordingly.

Our current plan is not to land the full Bolt backend directly in one step. Instead, we want to first split out the common-layer changes into a few smaller PRs (such as #12432, #12437, #12435, #12433 and #12431), so they can be reviewed firstly and make sure they do not affect the existing Velox / CH backends.

PR #12454 is the Bolt backend part built on top of those common patches. We’d like the common patches to be discussed first, and then continue the discussion on whether Bolt itself should be accepted as one of the supported backends in Gluten through the normal dev discussion / vote process.

[jackylee-ch]

@jackylee-ch Thanks for pointing this out.

We have created a tracking issue here: #12456, and I also updated the PR description in #12454 accordingly.

Our current plan is not to land the full Bolt backend directly in one step. Instead, we want to first split out the common-layer changes into a few smaller PRs (such as #12432, #12437, #12435, #12433 and #12431), so they can be reviewed firstly and make sure they do not affect the existing Velox / CH backends.

PR #12454 is the Bolt backend part built on top of those common patches. We’d like the common patches to be discussed first, and then continue the discussion on whether Bolt itself should be accepted as one of the supported backends in Gluten through the normal dev discussion / vote process.

Great! Thanks for your work. Could you add the issue number to the PR title so we can track the changes? It would be better if there were sub-issues.

[taiyang-li]

@jackylee-ch yes, of course.

[taiyang-li]

@jackylee-ch thanks for your suggestions. I'd create tracking sub-issues under #12456. If there are any other places that need improvement, please let me know.

[jackylee-ch]

@jackylee-ch thanks for your suggestions. I'd create tracking sub-issues under #12456. If there are any other places that need improvement, please let me know.

It looks good so far. I’m happy that Gluten offers enhanced backend extensibility. As for the addition of the Bolt Backend in Stage 2, we still need to conduct a valid vote, taking into account the Apache community's processes.

[taiyang-li]

@jackylee-ch Thanks for the recognition.

For now, the Bolt backend PR in Stage 2 is mainly there as an entry point for discussion. It still depends on the Stage 1 common patches being finished and settled first. After Stage 1 is done, we will still make some necessary updates in Stage 2, mainly:

1. further reduce the changes in common code, so Bolt-specific parts do not affect the existing backends, and it is also easier for this PR to keep up with upstream;
2. finish the necessary adjustments and validations, and make sure CI and compatibility checks can pass stably.

Our plan is to first make the boundary between the common-layer changes and the Bolt backend itself clearer, and then keep improving it based on community feedback. When it is all ready, we will start a vote following the Apache community process.

[taiyang-li]

@marin-ma I had resolved all the comments. Could you help review it again?

[marin-ma]

LGTM. Thanks.