Connect Pool by ericm-db · Pull Request #30 · ericm-db/spark

ericm-db · 2026-07-08T17:53:07Z

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

github-actions · 2026-07-08T17:53:19Z

⚠️ Pull Request Title Validation

This pull request title does not contain a JIRA issue ID.

Please update the title to either:

Include a JIRA ID: [SPARK-12345] Your description
Mark as minor change: [MINOR] Your description

For minor changes that don't require a JIRA ticket (e.g., typo fixes), please prefix the title with [MINOR].

This comment was automatically generated by GitHub Actions

…installed NumPy ### What changes were proposed in this pull request? SPARK-58553 replaced the `pandas_udf`-based `np.fmax` / `np.fmin` implementations in the pandas API on Spark with native `F.greatest` / `F.least` expressions. When the two operands are equal (for example `+0.0` and `-0.0`), the native mapping breaks the tie by returning the **first** operand. NumPy changed this signed-zero tie-break at **2.3.0**: `>= 2.3.0` returns the first operand, while older versions return the **second**. The native mapping therefore matches NumPy `>= 2.3.0` but disagrees with older versions on the sign of a `±0.0` result. This PR selects the tie operand based on the installed NumPy version so the result matches `np.fmax` / `np.fmin` on that NumPy: return the first operand on `>= 2.3.0`, the second on older versions. The implementation stays fully native (`F.greatest` / `F.least`); only which operand is returned on a signed-zero tie differs by version. **Why not restore the original `pandas_udf` fallback for old NumPy?** The original UDF matched the installed NumPy automatically (it calls `np.fmax` in a Python worker), so restoring it for `< 2.3.0` would also be correct. But that reintroduces the per-batch JVM <-> Python round trip that SPARK-58553 removed, losing the performance and optimizer benefits for those users. Since the signed-zero tie is the **only** cross-version difference (verified exhaustively over every combination of `{-inf, -2, -1, -0.0, +0.0, 1, 2, inf, nan}` from NumPy 1.23.2 through 2.4.1 — the tie-break flips at 2.3.0 and nothing else changes), and NumPy `< 2.3.0` is a frozen release range, selecting the matching tie operand keeps the native fast path while producing results identical to `np.fmax` / `np.fmin`. **Scope:** `fmax` / `fmin` is the only affected function. The full `test_numpy_compat.py` suite (18 tests, including the generic mapping sweeps and every other SPARK-58532 conversion — `fmod`, `ldexp`, `heaviside`, `reciprocal`, `float_power`, bitwise shifts, `signbit`, etc.) passes on the minimum dependencies; the signed-zero tie in `fmax` / `fmin` is the only version-sensitive behavior. ### Why are the changes needed? The scheduled "Build / Python-only (Minimum dependencies of PySpark)" build (NumPy 1.23.2) fails `pyspark.pandas.tests.test_numpy_compat NumPyCompatTests.test_np_fmax_fmin`. The test asserts the sign bit of the result via `np.signbit`, and on the two `±0.0` tie rows the native mapping (first operand) disagrees with the reference computed from the installed NumPy (second operand on 1.23.2). Regular CI runs a newer NumPy (`>= 2.3.0`), where the native choice matches, which is why the original change passed pre-merge CI and the failure only surfaced in the minimum-dependency build. ### Does this PR introduce _any_ user-facing change? No. There is no change relative to any released Spark version (the released implementation used the `pandas_udf`, which already matched the installed NumPy). This aligns the unreleased native implementation from SPARK-58553 with `np.fmax` / `np.fmin` on NumPy `< 2.3.0`. The numeric value is unchanged in all cases (`+0.0` and `-0.0` are numerically equal); only the sign bit of a zero result on a `±0.0` tie is corrected to match the installed NumPy. ### How was this patch tested? - `pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_fmax_fmin` and the full `NumPyCompatTests` suite (18 tests) pass on NumPy 2.4.1 and in a minimum-dependency environment (NumPy 1.23.2, pandas 2.2.0, pyarrow 18.0.0) that reproduces the scheduled build. - Confirmed the failure reproduces on NumPy 1.23.2 without this change and is resolved with it. - Verified across installed NumPy wheels (1.23.2 through 2.4.1) that the signed-zero `±0.0` tie is the only `fmax` / `fmin` behavior that differs between versions, and that the tie-break flips at exactly 2.3.0. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) Closes apache#57978 from Spenserrrr/numpy-fmax-fmin-version-gate. Authored-by: Spenser Sun <hsun112358@gmail.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

### What changes were proposed in this pull request? Adds golden-file drift tests for `pa.Table.to_pandas()` under the SPARK-54936 umbrella. New file `python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py` with two classes: - `PyArrowTableToPandasDefaultTests` -- multi-column DataFrame assembly and the empty-table edges (0-column and 0-row) under a bare `to_pandas()`. - `PyArrowTableToPandasCoerceTemporalTests` -- `coerce_temporal_nanoseconds=True` with `date_as_object` at its default and `=False`, plus timestamp/duration overflow rows. Spark calls `pa.Table.to_pandas()` in only two places: the 0-column early return in `python/pyspark/sql/pandas/conversion.py` (which passes `coerce_temporal_nanoseconds` and `date_as_object`) and a bare whole-Table conversion of a SQL command result in `python/pyspark/sql/connect/client/core.py`. Those are the only arguments Spark ever passes to a Table, which is why the tests cover just the bare call and those two flags; every other `to_pandas` flag (`self_destruct`/`split_blocks`/`use_threads`, `types_mapper`, `integer_object_nulls`, `zero_copy_only`) is applied only to `Array`/`ChunkedArray` columns, never to a whole Table. `Table.to_pandas()` returns a `DataFrame` while `Array`/`ChunkedArray.to_pandas()` return a `Series`, so the golden files have to render two shapes: - The input-table anchor column needs a `pa.Table` renderer, so `repr_value` in the shared `python/pyspark/testing/goldenutils.py` gains a `pa.Table` branch (`repr_arrow_table_value`). Without it a `pa.Table` fell through to the generic `str(value)` formatter, which is verbose and, being PyArrow's own `__repr__`, could change across versions and produce spurious golden diffs. The new helper renders each column with PyArrow's stable scalar formatting plus the Arrow schema. This is the only change to the shared file. - The converted-DataFrame result column is formatted by a small helper local to the test (`_repr_dataframe`), which serializes per-column via `tolist()`. It deliberately does not go through the shared `repr_pandas_value` (which uses `to_json()`): `to_json()` defaults to an epoch-nanosecond date format, so it raises `OverflowError` on out-of-nanosecond-range datetimes -- e.g. the year-9999 date rows, which under the default `date_as_object=True` come back as an object column of Python `datetime.date` objects that `to_json` overflows converting to epoch nanoseconds -- and it renders in-range dates as opaque epoch integers. `tolist()` is overflow-safe and keeps readable Python-native values, matching how `repr_pandas_series_value` already formats a Series. Keeping it local leaves the shared `repr_pandas_value` -- and the committed goldens of the existing `test_pandas_udf_return_type` coercion test that depend on its `to_json()` format -- untouched. ### Why are the changes needed? Part of SPARK-54936, which pins upstream PyArrow / pandas conversion behavior so that a library version bump changing it fails loudly in CI instead of silently returning wrong data. `pa.Table.to_pandas` is the last `to_pandas` conversion Spark relies on without golden coverage (`Array` and `ChunkedArray` are already covered; a `RecordBatch` is converted column-by-column rather than via `RecordBatch.to_pandas`). ### Does this PR introduce _any_ user-facing change? No. Test-only, plus one test-infrastructure helper in `goldenutils.py` (the `pa.Table` renderer). ### How was this patch tested? New golden tests, validated across a PyArrow 18-25 x pandas 2/3 sweep (16 combinations), with version-legitimate differences recorded as `LooseVersion`-guarded `overrides`. The sweep is Linux/x86 only, so an ARM follow-up may be needed once CI runs the full matrix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code Closes apache#57939 from Spenserrrr/table-to-pandas-tests. Authored-by: Spenser Sun <hsun112358@gmail.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

### What changes were proposed in this pull request? Print a blank line before `merge_spark_pr.py` asks whether to update an associated JIRA. ### Why are the changes needed? After a merge summary is posted, the JIRA prompt currently appears immediately after the summary attribution: ``` *Posted by `merge_spark_pr.py`* Would you like to update an associated JIRA? (y/N): ``` The blank line makes the transition to the next interactive step easier to read. ### Does this PR introduce _any_ user-facing change? No. This only adjusts output formatting in the Spark committer merge tool. ### How was this patch tested? The following checks passed: ``` cd dev conda run -n spark-dev-313 python -m doctest merge_spark_pr.py ``` 79 doctests passed. ``` conda run -n spark-dev-313 ruff check --config pyproject.toml dev/merge_spark_pr.py conda run -n spark-dev-313 ruff format --check --config pyproject.toml dev/merge_spark_pr.py git diff --check ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) Closes apache#57984 from zhengruifeng/minor-merge-script-jira-prompt-spacing-dev1. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

### What changes were proposed in this pull request? Update `dev/merge_spark_pr.py` to skip the "Would you like to update an associated JIRA?" prompt when merging a MINOR PR. ### Why are the changes needed? MINOR PRs have no associated JIRA ticket, so there is nothing to resolve after merging. The script still prompts to update an associated JIRA, which is unnecessary and must be dismissed manually for every MINOR merge. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? `python3 -m doctest dev/merge_spark_pr.py` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Deepseek V4 Pro Closes apache#57983 from pan3793/skip-minor-jira-prompt. Authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

…the merge in merge_spark_pr.py ### What changes were proposed in this pull request? This PR changes how `dev/merge_spark_pr.py` handles a committer declining to resolve a conflicting backport cherry-pick. Previously, when a cherry-pick onto a maintenance branch conflicted, the script prompted `Would you like to manually fix-up this merge?`. Answering `N` went through `continue_maybe(..., cherry=True)`, which aborted the cherry-pick and then called `fail("Okay, exiting")` — terminating the whole run via `sys.exit(-1)`. As a result the committer never reached the JIRA-resolution step, and the process exited non-zero, even though the merge into the target branch (and any earlier cherry-picks) had already been pushed. This PR makes declining a cherry-pick fix-up skip only that one branch and continue: - Add a `SkipCherryPick` exception. When a cherry-pick fix-up prompt is declined, `continue_maybe` aborts the cherry-pick, restores the working tree (`clean_up()`), and raises `SkipCherryPick` instead of calling `fail()`. - `cherry_pick()` catches `SkipCherryPick` and returns only the picks that actually landed (empty, or — in the Upstream-First two-branch path — just the sibling branch that was already pushed). - The two merge/backport loops in `main()` already consume the returned list generically (`merged_refs + []` is a no-op), so no loop changes are needed: after a skip they simply offer the next branch and still proceed to resolve the associated JIRA. Hard aborts elsewhere are unchanged (e.g. declining the push prompt, or choosing `[a]bort` at the Upstream-First prompt, still exit). ### Why are the changes needed? Backport cherry-pick conflicts are routine, and by the time one occurs the merge into the target branch has already been pushed. Aborting the entire script on a declined fix-up means the committer: - skips JIRA resolution (the ticket is left Open, which is easy to miss and tedious to reconcile after the fact), and - gets a non-zero exit for what is a normal "do not backport that one branch" decision. Declining a single conflicting backport should skip just that branch and let the merge finish cleanly. ### Does this PR introduce _any_ user-facing change? No. This changes a committer-only developer tool (`dev/merge_spark_pr.py`); it is not part of any Spark release artifact. ### How was this patch tested? - The module's inline doctests still pass (run by `doctest.testmod()` at startup): 76 attempted, 0 failed. - `python3 -m py_compile dev/merge_spark_pr.py` is clean. - Manually traced the affected control-flow paths: - single-branch pick declined -> `cherry_pick` returns `[]`, the loop re-prompts and JIRA resolution still runs; - Upstream-First `[b]oth` path with the second pick declined -> the already-pushed sibling branch is still returned and recorded in the merge comment / `merged_refs`; - declining non-cherry prompts, and `[a]bort` at the Upstream-First prompt, still hard-exit as before. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code with Claude Opus 4.8 Closes apache#57959 from uros-b/merge-script-cherrypick-skip. Authored-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

…_TEMP_1081 ### What changes were proposed in this pull request? Assign the name `TABLE_LOCATION_URI_NOT_SPECIFIED` to the legacy error condition `_LEGACY_ERROR_TEMP_1081`. This error is raised by `QueryCompilationErrors.tableNotSpecifyLocationUriError`, called from `CatalogTable.location`, when a table's storage `locationUri` is absent. The new condition is given SQLSTATE `42601`; the message text and the `identifier` parameter are unchanged. ### Why are the changes needed? The error conditions [README](https://github.com/apache/spark/blob/master/common/utils/src/main/resources/error/README.md) disallows new `_LEGACY_ERROR_TEMP_*` entries and asks existing ones to be assigned proper names. This resolves one of them, under the umbrella [SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935). ### Does this PR introduce _any_ user-facing change? No. The `_LEGACY_ERROR_TEMP_*` names are not part of the public API. The error condition name changes to `TABLE_LOCATION_URI_NOT_SPECIFIED` and now reports SQLSTATE `42601`; the rendered message ("Table `<identifier>` did not specify locationUri.") is unchanged. ### How was this patch tested? Added a `checkError` test in `QueryCompilationErrorsSuite` that constructs a `CatalogTable` with empty storage and asserts the `TABLE_LOCATION_URI_NOT_SPECIFIED` condition when `location` is accessed. `SparkThrowableSuite` validates error-file formatting/sorting and that non-legacy conditions carry a SQLSTATE. ### Was this patch authored or co-authored using generative AI tooling? Yes. Closes apache#57537 from gatorsmile/SPARK-58349-table-location-uri. Authored-by: Xiao Li <lixiao@databricks.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

…R_TEMP_3014 ### What changes were proposed in this pull request? Assign the name `EMPTY_COLLECTION_NOT_ALLOWED` to the legacy error condition `_LEGACY_ERROR_TEMP_3014`. This error is a `SparkUnsupportedOperationException` raised by `SparkCoreErrors.emptyCollectionError`, thrown by RDD reduce/first-style operations on an empty RDD (`RDD.scala`). The new condition is given SQLSTATE `0A000`; the message text ("empty collection") and its parameterless shape are unchanged. ### Why are the changes needed? The error conditions [README](https://github.com/apache/spark/blob/master/common/utils/src/main/resources/error/README.md) disallows new `_LEGACY_ERROR_TEMP_*` entries and asks existing ones to be assigned proper names. This resolves one of them, under the umbrella [SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935). ### Does this PR introduce _any_ user-facing change? No. The `_LEGACY_ERROR_TEMP_*` names are not part of the public API. The error condition name changes to `EMPTY_COLLECTION_NOT_ALLOWED` and now reports SQLSTATE `0A000`; the underlying message text ("empty collection") is unchanged. ### How was this patch tested? Updated the "empty RDD" test in `RDDSuite` from a `getMessage.contains` assertion to `checkError` on the `EMPTY_COLLECTION_NOT_ALLOWED` condition. Other RDD tests that hit this path assert only the exception type (`UnsupportedOperationException`), which still holds since `SparkUnsupportedOperationException` extends it. `SparkThrowableSuite` validates SQLSTATE presence and error-file formatting/sorting for the new entry. ### Was this patch authored or co-authored using generative AI tooling? Yes Closes apache#57539 from gatorsmile/SPARK-58351-empty-collection. Authored-by: Xiao Li <lixiao@databricks.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

…_TEMP_2070 ### What changes were proposed in this pull request? Assign the name `WRITING_JOB_FAILED` to the legacy error condition `_LEGACY_ERROR_TEMP_2070`. This error is a `SparkException` raised by `QueryExecutionErrors.writingJobFailedError`, thrown by `WriteToDataSourceV2Exec` when a DataSource V2 batch write fails and the subsequent `abort()` also fails (the original cause is chained onto the exception). The new condition is given SQLSTATE `58030` (matching the sibling `TASK_WRITE_FAILED`); it stays parameterless and preserves the chained `cause`. ### Why are the changes needed? The error conditions [README](https://github.com/apache/spark/blob/master/common/utils/src/main/resources/error/README.md) disallows new `_LEGACY_ERROR_TEMP_*` entries and asks existing ones to be assigned proper names. This resolves one of them, under the umbrella [SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935). ### Does this PR introduce _any_ user-facing change? No. The `_LEGACY_ERROR_TEMP_*` names are not part of the public API. The error condition name changes to `WRITING_JOB_FAILED` and now reports SQLSTATE `58030`; the message text ("Writing job failed.") is unchanged. ### How was this patch tested? Added a `checkError` test in `DataSourceV2Suite` with a new `CommitAndAbortFailingDataSource` whose batch write fails to commit and then fails to abort, driving the exact `WriteToDataSourceV2Exec` branch that raises `WRITING_JOB_FAILED`. `SparkThrowableSuite` validates SQLSTATE presence and error-file formatting/sorting for the new entry. ### Was this patch authored or co-authored using generative AI tooling? Yes This patch had conflicts when merged, resolved by Committer: Cheng Pan <chengpan@apache.org> Closes apache#57540 from gatorsmile/SPARK-58352-writing-job-failed. Authored-by: Xiao Li <lixiao@databricks.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

### What changes were proposed in this pull request? This pr upgrade janino from 3.1.9 to 3.1.12. ### Why are the changes needed? Janino 3.1.9 and earlier are affected by [CVE-2023-33546](https://nvd.nist.gov/vuln/detail/cve-2023-33546), a disputed DoS issue where deeply nested user-supplied input can trigger a `StackOverflowError` in Janino parser APIs such as `ExpressionEvaluator.guessParameterNames`. Although Spark SQL codegen uses Janino through `ClassBodyEvaluator` rather than that specific API, upgrading removes the vulnerable dependency version reported by dependency scanners and picks up Janino's parser/codegen robustness fixes. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#57266 from zml1206/SPARK-58134. Authored-by: Mingliang Zhu <zhuml1206@gmail.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>

…e ML guide ### What changes were proposed in this pull request? Removes the `Highlights in 3.0` section from `docs/ml-guide.md`, and rewrites a sentence that still described DataFrame-based API feature parity as future work in the Spark 2.x releases. ### Why are the changes needed? Both are stale on the current codebase. The guide keeps no equivalent `Highlights` section for 1.x or 2.x, so dropping superseded highlights is the established practice: SPARK-30934 replaced the 2.3 highlights with the 3.0 ones rather than accumulating them. The `Migration Guide` section immediately below already points readers to the archived migration guide, and the 3.0 features remain documented on their own pages, so no user-facing information is lost. Feature parity between the DataFrame-based and RDD-based APIs was reached at Spark 2.3, as `ml-pipeline.md` states, so the future-tense sentence is corrected to past tense. ### Does this PR introduce _any_ user-facing change? No, other than the updated documentation. ### How was this patch tested? Documentation only. No internal doc links point at the removed section's anchor, and the surrounding sections remain well-formed after the deletion. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes apache#57987 from uros-b/doc-mlguide-stale-highlights. Authored-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

### What changes were proposed in this pull request? Fixes four grammar errors in build-file comments: - `sql/catalyst/pom.xml` and `sql/core/pom.xml`: `However, a closed due to "Cannot Reproduce" Maven bug` becomes `However, a Maven bug closed as "Cannot Reproduce"`. - `sql/core/pom.xml`: `dependencies are needed for maven test builds on later hadoop releases` becomes `dependencies is needed for Maven test builds on later Hadoop releases` (the subject is the singular "declaration"; also fixes proper-noun casing and a missing space before the comment close). - `project/SparkBuild.scala`: `There are a reasons` becomes `There are reasons`. ### Why are the changes needed? These comments explain non-obvious build decisions, so they are worth keeping readable. The first is duplicated verbatim across the two POMs, which is how the error propagated. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Comment text only: Maven and SBT never parse it, so there is no build impact, and no version, coordinate, scope, or ordering is touched. Both POMs were confirmed to still parse as well-formed XML. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes apache#57988 from uros-b/pom-build-comment-grammar. Authored-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

…umnStat ### What changes were proposed in this pull request? Unbraces 14 `${colName}` interpolations to `$colName` in `CatalogColumnStat.toMap`/`fromMap`, leaving the qualified `${CatalogColumnStat.KEY_...}` / `${KEY_...}` selects braced. ### Why are the changes needed? The two methods mix `${colName}` and the bare `$colName` for the same value, including on adjacent lines: the histogram key already reads `s"$colName.${CatalogColumnStat.KEY_HISTOGRAM}"`. This makes all 14 remaining sites consistent with that style. The qualified selects keep their braces on purpose, because unbracing `${CatalogColumnStat.KEY_VERSION}` would interpolate the object and append `.KEY_VERSION` as literal text. ### Does this PR introduce _any_ user-facing change? No. These strings are the on-disk table-property keys for column statistics. In every case `colName` is a simple parameter immediately followed by `.`, so `$colName.` parses identically to `${colName}.` and the generated keys are byte-for-byte unchanged, preserving metadata compatibility. ### How was this patch tested? Existing catalog statistics tests cover `toMap`/`fromMap`. The change is a lexical simplification with identical output. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes apache#57989 from uros-b/consistency-catalog-colstat-interp. Authored-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

…rences in release tooling ### What changes were proposed in this pull request? Three fixes to developer and release tooling: - `dev/create-release/release-build.sh`: removes the dead `lsof` band-aid block, and fixes a typo in the usage help (`reposiotry` to `repository`). - `.github/PULL_REQUEST_TEMPLATE`: updates the `ConfigEntry.scala` path to its current location under `common/utils`. ### Why are the changes needed? The `lsof` block assigns `$LSOF` and probes for the binary, but `$LSOF` is never used anywhere in the script or in anything it sources or that calls it, so the SPARK-22377 workaround has had no effect since the Jenkins machines it targeted were retired. The PR template points contributors at `core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala`, which no longer exists; the file now lives under `common/utils`. ### Does this PR introduce _any_ user-facing change? No. Developer and release tooling only. ### How was this patch tested? No functional change. `release-build.sh` still parses (`bash -n`), `$LSOF` was confirmed to have no reference outside the removed block (including in `release-util.sh` and `do-release.sh`), and the corrected `ConfigEntry.scala` path exists on master while the old one does not. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes apache#57972 from uros-b/devexp-stale-paths-and-dead-lsof. Authored-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

…s during analysis ### What changes were proposed in this pull request? SPARK-58389 changed DSv2 relation caching to include all read options. This is required to preserve each reference's complete option bag, but it also means references that differ only in scan-specific options can independently call `loadTable` and receive different concrete table versions within one query. This PR separates relation reuse from table-state pinning: - The existing `relationCache` remains keyed by all read options and reuses a finalized relation only when the complete option bags match. - A new query-scoped `tableCache` is keyed by catalog, identifier, time travel, and table-state options. References with the same table-state key reuse one concrete `Table`, while retaining their own complete options in their relations. - References with different table-state options do not share a table pin. - `sharedRelationCache` and CacheManager reuse match table identity and table-state options. The resolved relation still retains the current reference's complete option bag. This PR adds `tableStateOptionKeys()` directly to the evolving `TableCatalog` API so a catalog can declare which raw options may affect the table state selected by `loadTable`, such as a branch, tag, snapshot, or version. Spark passes only those declared options to the options-aware `loadTable`. The default implementation declares no table-state options, so catalogs that do not override it receive an empty option map during table loading. The complete option map remains on the resolved relation for scan and write planning. While applying the same table-pinning model to cacheable `V2TableReference` resolution, this PR also fixes two existing gaps in `getOrLoadRelation`: - It previously called `loadTable(identifier)` without passing the options captured in the table reference. A table-cache miss now uses the options-aware catalog API with only the reference's declared table-state options. - It previously did not consult `sharedRelationCache`. Temporary-view re-resolution now consults `sharedRelationCache` while establishing the initial table pin, allowing it to preserve a `Table` already pinned through CacheManager. Transaction references still use the `Table` loaded through the transaction catalog and do not consult `sharedRelationCache`. Write targets remain non-cacheable and bypass the query-scoped read caches. The resulting resolution flow is: 1. Check `relationCache` using the full-option relation key. 2. On a miss, check `tableCache` using the table-state key. 3. On a `tableCache` hit, construct a relation from the pinned `Table` and the current reference's complete options. Do not call `loadTable` or consult `sharedRelationCache`. 4. On a `tableCache` miss, load the current `Table` through the applicable path. An options-aware `loadTable` receives only the declared table-state options. 5. Where `sharedRelationCache` lookup applies, reuse its table only when table identity and table-state options match. The shared cached `Table` establishes the initial pin on a match; otherwise, the newly loaded `Table` establishes it. 6. Store the pinned `Table` in `tableCache` and the finalized relation in `relationCache`. Execution-time table refresh uses the same table-state option projection to preserve this first-resolution-wins behavior. ### Why are the changes needed? A catalog may accept both table-state options and scan-specific options. Using the complete option bag for relation reuse is necessary, but using it as the only level of caching can cause references in the same table-state domain to load different concrete table versions during one query. The new `tableCache` pins one concrete `Table` per state key while finalized relations remain distinct when their complete option bags differ. Shared cache and CacheManager lookup use the same table-state projection while preserving the current reference's complete options. Forwarding declared table-state options from `V2TableReference` is necessary because those options may select the table state being reloaded. Temporary views additionally need the `sharedRelationCache` bridge to preserve a CacheManager-pinned `Table`. Different state domains, including different parsed time-travel specifications, continue to resolve and pin independently. ### Does this PR introduce _any_ user-facing change? Yes, for catalog implementors only. This adds the default `tableStateOptionKeys()` method directly to the evolving `TableCatalog` API. Catalogs whose raw options select table state can override this method together with the options-aware `loadTable`. By default, no options are considered table-state options and `loadTable` receives an empty option map. Complete options are still retained for subsequent scan and write planning. There is no new SQL syntax or configuration. The option-forwarding behavior refined here was introduced on the unreleased master branch by SPARK-58389. ### How was this patch tested? Added regression coverage for table-state projection and load-option filtering, table pinning, default no-state-option behavior, `sharedRelationCache` and CacheManager matching, nested analysis, execution refresh, and temporary-view, transaction, and write-target `V2TableReference` behavior. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex Closes apache#57799 from yyanyy/yan-yan_data/spark-dsv2-table-state-cache-20260804. Authored-by: yyanyy <yyanyyyy@gmail.com> Signed-off-by: Anton Okolnychyi <aokolnychyi@apache.org>

… bind addresses ### What changes were proposed in this pull request? Teach Spark to distinguish the address a Kubernetes driver **listens on** from the address executors should **connect to** when direct driver-Pod-IP mode is enabled. Normally, using the driver's bind address as its advertised address works: a Kubernetes-managed driver binds directly to its Pod IP, and executors can connect to that IP without going through a driver Service. That assumption breaks when a driver intentionally binds to every network interface. An address such as `0.0.0.0` or `::` means "listen everywhere," but it does not identify a reachable peer. This change makes the choice depend on the address itself. A concrete bind address continues to use the existing direct-Pod-IP path. A wildcard bind address instead preserves the separately configured, routable `spark.driver.host`. Both driver initialization and executor endpoint construction apply the same rule, so the address advertised by the driver and the address given to executors remain consistent. Wildcard recognition supports IPv4 and IPv6 without performing DNS lookups, and existing IPv6 normalization is preserved. ### Why are the changes needed? [SPARK-58748](https://issues.apache.org/jira/browse/SPARK-58748) Spark deliberately separates two network settings: ```properties # Where the driver opens its listening sockets. spark.driver.bindAddress=0.0.0.0 # The reachable address advertised to executors. spark.driver.host=10.0.0.42 ``` This is a legitimate and common configuration for Kubernetes client-mode applications and Spark Connect servers: the process listens on all local interfaces, while executors are told to connect to the driver's actual Pod IP or a routable Service hostname. The problem appears when direct driver-Pod-IP mode is enabled: ```properties spark.master=k8s://https://kubernetes.example:6443 spark.driver.bindAddress=0.0.0.0 spark.driver.host=10.0.0.42 spark.kubernetes.executor.useDriverPodIP=true ``` Spark currently assumes the driver's bind address is always its Pod IP. Consequently, `SparkContext` replaces the configured advertised host with `0.0.0.0`, and Kubernetes executor creation independently uses that same wildcard to construct the scheduler endpoint: ```text Before Driver listens on: 0.0.0.0:7078 Advertised driver host: 0.0.0.0 Executor driver URL: spark://CoarseGrainedScheduler0.0.0.0:7078 Result: executors cannot connect or register After Driver listens on: 0.0.0.0:7078 Advertised driver host: 10.0.0.42 Executor driver URL: spark://CoarseGrainedScheduler10.0.0.42:7078 Result: executors connect normally ``` The driver starts successfully because binding to `0.0.0.0` is valid. The failure only becomes visible when executors try to register, leaving the application running but unable to execute tasks. Fixing only one side is insufficient: the driver and executor each make their own address-selection decision, and both must preserve the advertised host. The same distinction applies to IPv6. For example, a driver may listen on `::` while advertising a concrete address such as `[2001:db8::42]`; executors must receive the concrete address, never the IPv6 wildcard. Compressed, bracketed, and expanded IPv6 wildcard forms are handled consistently. Spark 4.1 and 4.2 are affected when `spark.kubernetes.executor.useDriverPodIP` is explicitly enabled. Spark 4.3 and current master enable it by default, so previously valid wildcard-bind configurations can fail without any application-level configuration change. ### Does this PR introduce _any_ user-facing change? Yes. Kubernetes applications that bind their driver to all interfaces now retain the reachable address configured in `spark.driver.host`, allowing their executors to register and run. This restores the existing documented distinction between the driver's listening address and its advertised address. Applications that bind directly to a concrete Pod IP continue using direct-IP routing, including existing IPv6 normalization. Applications with direct driver-Pod-IP mode disabled, and applications outside Kubernetes, retain their existing behavior. No configuration defaults or public APIs change. ### How was this patch tested? ```bash build/sbt -Pkubernetes \ 'core/testOnly org.apache.spark.SparkContextSuite -- -z "SPARK-58748"' \ 'core/testOnly org.apache.spark.util.UtilsSuite -- -z "SPARK-58748"' \ 'kubernetes/testOnly org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStepSuite' \ 'core/scalastyle' \ 'core/Test/scalastyle' \ 'kubernetes/scalastyle' \ 'kubernetes/Test/scalastyle' ``` All 41 tests passed: one targeted `SparkContextSuite` regression, one targeted `UtilsSuite` regression, and all 39 `BasicExecutorFeatureStepSuite` tests. Together, they cover default-enabled driver initialization, explicitly enabled and disabled executor configuration, IPv4 and IPv6 wildcard addresses, concrete IPv4 and IPv6 addresses, executor driver URLs, and existing IPv6 normalization. All four Scala style checks also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex Closes apache#57977 from sunchao/dev/chao/codex/spark-k8s-wildcard-driver-address-oss. Authored-by: Chao Sun <chao@openai.com> Signed-off-by: Chao Sun <chao@openai.com>

…hState init failure ### What changes were proposed in this pull request? This patch makes the driver-side `TransformWithStateInPySpark` pre-initialization runner clean up on every lifecycle path. It moves `init()` and `process()` into `Utils.tryWithSafeFinally`, guaranteeing that `stop()` runs when initialization or processing fails. It also preserves the original initialization failure when cleanup itself throws by relying on `tryWithSafeFinally`'s suppressed-exception behavior. The runner's state-server daemon thread is nullable until the end of initialization, so `stop()` now checks for null before interrupting it. A six-case regression suite covers initialization failure cleanup, repeated failures, suppressed cleanup errors, process failure wrapping, the success path, and stopping before state-server startup. JIRA: https://issues.apache.org/jira/browse/SPARK-58751 ### Why are the changes needed? `StreamingPythonRunner.init()` creates the Python worker before it finishes initialization. If initialization fails afterward, the previous code never reached `stop()`, leaking the worker and its associated resources for the driver's lifetime. Repeated streaming restarts can accumulate these leaked resources and eventually prevent new isolated workers from starting. ### Does this PR introduce _any_ user-facing change? No. Successful execution behavior is unchanged. This only restores cleanup on existing failure paths and preserves the original initialization error instead of allowing cleanup failures or a null-thread error to obscure it. ### How was this patch tested? Added `TransformWithStateInPySparkPreInitCleanupSuite` with six tests and no real Python worker dependency. Static validation passed with `git diff --check` and the changed-file line-length check. The focused Spark test could not run because this checkout does not have `sbt` installed and its launcher download was unavailable. A Maven `test-compile` fallback also could not resolve dependencies because the configured Maven mirrors were unreachable from the environment. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (Codex CLI). Closes apache#57941 from jon-gao-db/fix/transform-with-state-pre-init-cleanup. Lead-authored-by: jon-gao-db <242085654+jon-gao-db@users.noreply.github.com> Co-authored-by: Jonathan Gao <jonathan-gao@hotmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>

…ework ### What changes were proposed in this pull request? Add smart test selection for pyspark test framework. The approach is: 1. In `run-tests.py`, check changed files. Rule out the known irrelevant files (that belongs to `dev_tool` module). If the rest only contains `pyspark` files, and we are running CI for post-merge of apache/spark, trigger this mechanism by a. get the relevant changed file list b. write that list to a temp file c. pass the path to the temp file to `python/run-tests.py` 2. In `python/run-tests.py`, if changed file path is received, add that to the env var so all the subprocesses get it. 3. For all PySpark tests, in `setUpClass`, try to read that env var (which is super cheap). If the env var exists, get the list, build a pyspark graph (sub-second cost per module) and analyze if the module of the test class is impacted by the changed files. 4. If the test is not impacted, skip it. ### Why are the changes needed? We will only run impacted tests on python-only changes by accurately analyzing the changed files. This will save us plenty of CI time. ### Does this PR introduce _any_ user-facing change? In theory, `pyspark/testing` is public to users. However, this mechanism is for pyspark tests and is gated by the env var. It should not impact any users that do not want it. ### How was this patch tested? A feature test file is added for multiple cases. Some manual tests are done locally. However, whether it works for E2E scenarios, we need to observe the post-merge CIs for python-only changes. The changed code is also reviewed by Claude Code (Opus 4.8) ### Was this patch authored or co-authored using generative AI tooling? Only the test file is generated by LLM (with some tuning so it's better organized). Closes apache#57587 from gaogaotiantian/changed-files-aware. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Tian Gao <gaogaotiantian@hotmail.com>

### What changes were proposed in this pull request? Add an end-to-end test to `TimestampNanosFunctionsSuiteBase` (run in both ANSI modes) asserting that mode returns the most-frequent value keyed on the full nanos value (the frequent and rare values differ only within the microsecond) and preserves the input precision and family (NTZ/LTZ). Also add a deterministic golden SQL case (`mode(...)` with a unique most-frequent value) to `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql`. ### Why are the changes needed? Extend test coverage for timestamp nanosecond precision datatype. ### Does this PR introduce _any_ user-facing change? No, test only change. ### How was this patch tested? Test only change. ### Was this patch authored or co-authored using generative AI tooling? Co-Authored-By: Claude Code 4.8 Closes apache#57966 from stevomitric/stevomitric/nanos-mode-tests. Authored-by: Stevo Mitric <stevomitric2000@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

…anosecond-timestamp columns ### What changes were proposed in this pull request? Add a `HiveDDLSuite` test that creates a table with `TIMESTAMP_NTZ(9)` / `TIMESTAMP_LTZ(7)` columns while the nanosecond-timestamp preview flag is on (the default in tests via `Utils.isTesting`), then asserts that: - `DESCRIBE TABLE` renders `timestamp_ntz(9)` / `timestamp_ltz(7)`; - `SHOW CREATE TABLE` renders the parseable, uppercased `TIMESTAMP_NTZ(9)` / `TIMESTAMP_LTZ(7)`; - the emitted `SHOW CREATE TABLE` DDL re-parses and re-creates an identical nanos schema (round-trip). ### Why are the changes needed? SPARK-57835 added DESCRIBE / SHOW CREATE coverage only for the read-through path (table created with the flag on, introspected with the flag off). The basic happy path -- introspecting a nanos table with the flag on -- was still untested. This closes that DDL-introspection coverage gap for SPARK-56822. ### Does this PR introduce any user-facing change? No. Test-only. ### How was this patch tested? `build/sbt 'hive/testOnly org.apache.spark.sql.hive.execution.HiveDDLSuite'` (new test green). Non-vacuity confirmed by temporarily breaking the expected rendered value and observing the assertion fail. ### Was this patch authored or co-authored using generative AI tooling? Co-Authored-By: Claude Opus 4.8 Closes apache#57991 from stevomitric/stevomitric/nanos-describe-showcreate-tests. Authored-by: Stevo Mitric <stevomitric2000@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

… over nanosecond-precision timestamp columns ### What changes were proposed in this pull request? Add end-to-end regression-lock coverage for `DISTINCT` over the nanosecond-precision timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`). New suite `TimestampNanosDistinctSuiteBase` (with `TimestampNanosDistinctAnsiOnSuite` / `TimestampNanosDistinctAnsiOffSuite` subclasses), mirroring `TimestampNanosJoinSuiteBase`. For NTZ and LTZ, across `p` in `[7, 9]` and both whole-stage-codegen modes, it asserts: - DISTINCT removes exact duplicates but keeps two values that share `epochMicros` and differ only within the microsecond, and keeps exactly one NULL (unlike an equi-join, where NULL never matches NULL); - a `UNION` of two different nanos precisions widens the column to the higher precision (`findWiderDateTimeType`) and preserves the distinction. Also extends the golden files `timestamp-ntz-nanos.sql` / `timestamp-ltz-nanos.sql` with a DISTINCT case. ### Why are the changes needed? DISTINCT over a nanosecond-timestamp column already works today -- it rides on the generic nanos hashing/equality implemented in SPARK-57103 (`Murmur3Hash` / `XxHash64` / `HiveHash` over the carrier's `epochMicros: Long` and `nanosWithinMicro: Short in [0, 999]`) -- but lacked dedicated coverage. These tests lock the regression: if the nanos hash or equality path is later broken, sub-microsecond-distinct values collapse and the tests fail loudly. No production change. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? `TimestampNanosDistinctAnsiOnSuite` / `TimestampNanosDistinctAnsiOffSuite` (16 tests, all pass) and the regenerated golden SQL files. ### Was this patch authored or co-authored using generative AI tooling? Co-Authored-By: Claude Opus 4.8 Closes apache#57993 from stevomitric/stevomitric/nanos-distinct-tests. Authored-by: Stevo Mitric <stevomitric2000@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

### What changes were proposed in this pull request? Replace the immutable Scala `Map[String, Int]` used by `CountVectorizerModel` for vocabulary lookup with a pre-sized `java.util.HashMap[String, Integer]`. Build and broadcast the dictionary once, then dereference the broadcast value and its size once per document instead of once per token. Register `java.util.HashMap` with Spark Kryo so this remains compatible with `spark.kryo.registrationRequired=true`. ### Why are the changes needed? `CountVectorizerModel.transform` performs one vocabulary lookup per input token. A local Java 17 benchmark with 262,144 String keys, 2 million lookups, and a 90% hit rate measured: - Scala immutable Map: 69.5 ns/lookup - Spark OpenHashMap: 38.3 ns/lookup - java.util.HashMap: 20.7 ns/lookup For the same vocabulary, Java HashMap used about 13% less deserialized heap than the immutable map, produced a 26% smaller JavaSerializer + LZ4 broadcast payload, and deserialized about 47% faster. OpenHashMap used less heap and deserialized faster than Java HashMap, but Java HashMap provides the best tradeoff for this lookup-heavy workload. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added strict-Kryo coverage for `java.util.HashMap` and ran: - `build/sbt "core/testOnly *KryoSerializerSuite -- -z \"basic types\""` - `build/sbt "mllib/testOnly *CountVectorizerSuite"` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) Closes apache#58278 from zhengruifeng/ml-countvectorizer-openhashmap-dev-4. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>

…eption when allowKeysSubsetOfPartitionKeys ### What changes were proposed in this pull request? `KeyedPartitioning.createShuffleSpec` now sorts the distinct projected partition keys (via `toGrouped`) so they follow the same natural ascending ordering as `GroupPartitionsExec`. ### Why are the changes needed? SPARK-56877 added a check in `PartitioningCollection.fromPartitionings` requiring all `KeyedPartitioning`s to share equal `partitionKeys`. In a storage-partitioned join whose join keys are a subset of the partition keys (e.g. a v2 table partitioned by `[dt, bucket(16, c1)]` joined on `c1`), with `spark.sql.sources.v2.bucketing.shuffle.enabled` enabled so only the non-keyed side is re-shuffled, the keyed side's projected keys are sorted by `GroupPartitionsExec` while `createShuffleSpec` kept them in first-occurrence order. The two sides then carry the same keys in different orders and the query fails with: ``` java.lang.IllegalArgumentException: requirement failed: All KeyedPartitionings in a PartitioningCollection must have equal partitionKeys ``` ### Does this PR introduce _any_ user-facing change? No by default. ### How was this patch tested? Added a regression test in `KeyGroupedPartitioningSuite` (`SPARK-58988: v2 bucketed table with subset join keys joining v1 table`) that reproduces the failure and passes with the fix. Also ran `KeyGroupedPartitioningSuite`, `EnsureRequirementsSuite`, `GroupPartitionsExecSuite`, and `ProjectedOrderingAndPartitioningSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code Closes apache#58273 from ulysses-you/SPARK-58988. Authored-by: Xiduo You <ulyssesyou18@gmail.com> Signed-off-by: Xiduo You <ulyssesyou@apache.org>

…boolean, and narrower integer columns ### What changes were proposed in this pull request? This is a follow-up of apache#57856 (SPARK-58646), which replaced the scalar pandas UDF fallback of `np.reciprocal` with native Spark expressions for non-floating-point inputs. That change routed every non-`float`/`double` dtype through a single integer branch that hard-codes the int64 minimum as the divide-by-zero sentinel and casts the quotient through `long`. This does not match the previous pandas UDF (`np.reciprocal` applied to the pandas `Series`) for decimals, booleans, and narrower integers: | dtype | input | previous UDF (`np.reciprocal` -> Double) | merged native (apache#57856) | |---|---|---|---| | int64 (bigint) | `0` | `-9.2e18` (int64 min) | `-9.2e18` (unchanged) | | int32 (int) | `0` | `-2147483648` (int32 min) | `-9.2e18` | | int8/int16 (tinyint/smallint) | `0` | `0` (numpy `1 // 0` does not overflow on narrow widths) | `-9.2e18` | | boolean | `False` | `0.0` (numpy promotes bool to int8: `True -> 1`, `False -> 0`) | `-9.2e18` | | decimal | `2.5` | `0.4` (numpy takes a true floating reciprocal) | `0.0` (truncated to long) | This PR restores parity: - Decimal inputs now flow through the floating-point reciprocal branch (`typeof` starts with `decimal`), since numpy computes a true reciprocal for them. A decimal `0` (which the old UDF could not handle -- `np.reciprocal(Decimal('0'))` raises `DivisionByZero`) now maps to `inf`, consistent with the floating-point branch. - The integer/boolean branch now picks the divide-by-zero sentinel by column width -- int32 minimum for `int`, int64 minimum for `bigint`, and `0` for the narrower widths (`tinyint`, `smallint`, and `boolean` promoted to int8) -- and casts through `long` so boolean and narrower integers can take part in the division. int64 columns, the only case exercised by the original PR, are unchanged. ### Why are the changes needed? The merged native expression regressed the observable pandas-on-Spark behavior for decimal, boolean, and narrower-integer columns relative to the pandas UDF it replaced. This restores parity so that `np.reciprocal` produces the same results as before across all supported dtypes. ### Does this PR introduce _any_ user-facing change? No. apache#57856 is unreleased (master only), so this only fixes an unreleased regression before it ships; there is no change relative to any released Spark version. ### How was this patch tested? Added `test_np_reciprocal_non_default_dtypes` in `python/pyspark/pandas/tests/test_numpy_compat.py`, inherited by the Spark Connect parity suite, asserting `np.reciprocal(psser)` equals `np.reciprocal(pdf)` for int8/int16/int32, boolean, and decimal columns (covering positive, negative, and the per-width zero-overflow sentinel). ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#58218 from Yicong-Huang/reciprocal-nonfloat-fix. Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>

### What changes were proposed in this pull request? Remove the dead and broken `AutoSerializer` class from `python/pyspark/serializers.py`, along with its two references in `python/pyspark/tests/test_serializers.py`. ### Why are the changes needed? `AutoSerializer` is dead and broken code: - It is not exported in `__all__` and has no production callers. - Its `loads()` is broken under Python 3: it reads `_type = obj[0]`, which yields an `int` rather than a `bytes` value like `b"M"`/`b"P"`, so the type check never matches and every call raises `ValueError`. - The only reference is a smoke test in `test_serializers.py` (an import plus a `hash()` call). ### Does this PR introduce _any_ user-facing change? No. The class is not part of the public API and is non-functional. ### How was this patch tested? Existing `python/pyspark/tests/test_serializers.py` still passes after removing the unused import and `hash()` line. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#58275 from Yicong-Huang/remove-dead-autoserializer. Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>

### What changes were proposed in this pull request? [SPARK-59016](https://issues.apache.org/jira/browse/SPARK-59016) expands the FunctionRegistry CHAR/VARCHAR leak inventory to cover additional arities and nested array, map, and struct shapes. The allowlist is shape-aware: legitimate collection pass-through calls are distinguished from scalar transforming calls such as `reverse(c)` and `concat(c, ...)`, which must return plain STRING. The original scalar inventory remains under parent [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794), while the expanded templates are covered by SPARK-59016. ### Why are the changes needed? The previous inventory used only seven argument templates, so a CHAR/VARCHAR leak at another arity or nested shape would not fail CI. A function-name-only allowlist could also hide an invalid scalar leak when another shape of the same function is a legitimate pass-through. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - `build/sbt "sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z inventoried"` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 Closes apache#58299 from srielau/serge-rielau_data/SPARK-58794-p1. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>

…ey order ### What changes were proposed in this pull request? `ShuffleExchangeExec.getPartitioner` built its `KeyGroupedPartitioner` from `KeyedPartitioning.toGrouped`, i.e. from the *sorted* partition keys. It now builds it from `partitionKeys` in the order the partitioning declares, and asserts the keys are unique. `KeyedShuffleSpec.canCreatePartitioning` additionally refuses an ungrouped partitioning, so a spec whose keys are not unique is never chosen as the template for shuffling the other child. ### Why are the changes needed? `partitionKeys` is a physical layout indexed by partition id: partition `i` holds key `partitionKeys(i)`. With `spark.sql.sources.v2.bucketing.shuffle.enabled`, `EnsureRequirements` shuffles only the non-keyed side of a storage-partitioned join and reuses the keyed side's `KeyedPartitioning` as the target (`KeyedShuffleSpec.createPartitioning`). Re-deriving the order in `getPartitioner` therefore contradicts the side the shuffle is supposed to co-partition with, and the join reads rows from partitions holding different keys on each side. Those keys are not always sorted. Two producers report them unsorted: - `UnionExec` concatenates its children's keys in child order, so children partitioned by `identity(id)` holding `[3, 4]` and `[1, 2]` merge to `[3, 4, 1, 2]`. - a narrowing projection through `PartitioningPreservingUnaryExecNode` projects the keys onto a subset of key positions, which preserves neither sortedness nor uniqueness. Where uniqueness is lost the result no longer satisfies a `ClusteredDistribution`, so `EnsureRequirements` interposes a `GroupPartitionsExec` that re-groups and sorts. Where only sortedness is lost the partitioning still satisfies, nothing is interposed, and the unsorted keys are the real layout of that side. The result is silently wrong. Nothing catches it: `createPartitioning` passes the keyed side's `partitionKeys` *reference* through unchanged, so `PartitioningCollection.fromPartitionings` interns it on `eq` and never runs the `require` added by SPARK-56877, and `ValidateRequirements` compares the same declared keys on both sides. To reproduce, two v2 tables partitioned by `identity(id)` that report one input partition per key, one holding ids `3, 4` and the other `1, 2`: ```sql CREATE TABLE testcat.ns.nt1 (id BIGINT, data STRING) PARTITIONED BY (id); CREATE TABLE testcat.ns.nt2 (id BIGINT, data STRING) PARTITIONED BY (id); INSERT INTO testcat.ns.nt1 VALUES (3, 'c'), (4, 'd'); INSERT INTO testcat.ns.nt2 VALUES (1, 'a'), (2, 'b'); CREATE TABLE t1 (id BIGINT, x STRING) USING parquet; INSERT INTO t1 VALUES (1, 'x'), (2, 'x'), (3, 'x'), (4, 'x'); SET spark.sql.sources.v2.bucketing.shuffle.enabled=true; SELECT u.id, u.data, t1.x FROM (SELECT * FROM testcat.ns.nt1 UNION ALL SELECT * FROM testcat.ns.nt2) u JOIN t1 ON u.id = t1.id; ``` keyed side (not re-grouped) partition 0 -> id 3 | 1 -> 4 | 2 -> 1 | 3 -> 2 shuffled side, before partition 0 -> id 1 | 1 -> 2 | 2 -> 3 | 3 -> 4 shuffled side, after partition 0 -> id 3 | 1 -> 4 | 2 -> 1 | 3 -> 2 Before the change the query returns no rows; the correct answer is four. The affected branches are `master`, `branch-4.x` and `branch-4.3`. `branch-4.2` sorts in `getPartitioner` too but has no producer of unsorted keys, and `branch-4.1` and `branch-4.0` do not sort at all -- their `uniquePartitionValues` only deduplicates. SPARK-52246 fixed the same symptom for a different trigger, where the join keys are a subset of the partition keys. This one needs no subset keys. It is also distinct from SPARK-58988, which fixed the projected branch of `createShuffleSpec`. There a `GroupPartitionsExec` *is* interposed on the keyed side and sorts it, so the shuffled side's declared keys have to be sorted to match, and the mismatch surfaced as a loud `require` failure rather than wrong rows. Both are needed: reverting either one leaves its own case broken. ### Does this PR introduce _any_ user-facing change? Yes, it fixes wrong results, but only for queries that opt into `spark.sql.sources.v2.bucketing.shuffle.enabled`, which is off by default. ### How was this patch tested? Added two tests to `KeyGroupedPartitioningSuite`, one per producer of unsorted keys: the union case above, and a narrowing projection that drops a partition column. Both fail without the change, with `Correct Answer - 4` against `Spark Answer - 0` and `Correct Answer - 2` against `Spark Answer - 0` respectively. The plan shape is identical either way (one shuffle, no `GroupPartitionsExec`), which the tests assert, so `checkAnswer` is what discriminates. The union test also re-checks the answer with adaptive execution on, since the plan-shape assertions require it off. Added a `ShuffleSpecSuite` test for the `canCreatePartitioning` gate, covering a grouped spec with unsorted keys (accepted) and an ungrouped one (refused); it fails if the new clause is removed. Ran `ShuffleSpecSuite`, `KeyGroupedPartitioningSuite`, `EnsureRequirementsSuite`, `GroupPartitionsExecSuite`, `ProjectedOrderingAndPartitioningSuite`, `DataSourceV2Suite`, `WriteDistributionAndOrderingSuite`, `ValidateRequirementsSuite`, `PlannerSuite` and `AdaptiveQueryExecSuite`: 595 tests, 0 failures. Also verified with a temporary assertion that no covered scenario reaches `getPartitioner` with duplicate partition keys, and that a duplicating narrowing projection is handled by a `GroupPartitionsExec` under both `allowKeysSubsetOfPartitionKeys` and `requireAllClusterKeysForDistribution`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code Closes apache#58311 from peter-toth/SPARK-59022-keyed-shuffle-declared-key-order. Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…Fix Version prompt when no version can be inferred ### What changes were proposed in this pull request? Two fixes in `dev/merge_spark_pr.py`, both on the path where Fix Version inference produces nothing: 1. Blank input at the Fix Version prompt with no default now exits with a message instead of re-prompting forever. 2. On an already-resolved JIRA, `already contains all inferred fix versions` is printed only when something was in fact inferred. Otherwise the committer reaches the prompt. ### Why are the changes needed? Take a backport PR opened directly against `branch-4.2`, merged at a moment when JIRA has no unreleased `4.2.x` version -- either the line is EOL, or `4.2.2` has not been created yet after `4.2.1` shipped. Inference finds nothing and warns the committer to enter a version manually. What follows is wrong in both directions. **The linked JIRA is already resolved by the earlier master merge**, which is the normal state for a backport. Before: ``` JIRA issue SPARK-58880 already has status 'Resolved' (Fixed) Check if the JIRA information is as expected (y/N): y Target version for branch-4.2 is not found on JIRA, it may be archived or not created. Skipping it. JIRA issue SPARK-58880 already contains all inferred fix versions; no update needed. ``` The ticket has `['5.0.0']` and nothing was inferred, so the last line is false and contradicts the warning above it. The prompt never appears, and the merge records no Fix Version. After: ``` Target version for branch-4.2 is not found on JIRA, it may be archived or not created. Skipping it. JIRA issue SPARK-58880 has fix version(s) ['5.0.0']; no additional fix version could be inferred. Enter comma-separated additional fix version(s) []: No fix version entered; update SPARK-58880 manually. ``` **The JIRA is not yet resolved.** The committer reaches the prompt with an empty default, and pressing Enter never escapes, because `""` can never match a known version -- even though the retry message offers exactly that. Before: ``` Target version for branch-4.2 is not found on JIRA, it may be archived or not created. Skipping it. Enter comma-separated fix version(s) []: Specified version(s) [] not found in the available versions, try again (or leave blank and fix manually). Enter comma-separated fix version(s) []: Specified version(s) [] not found in the available versions, try again (or leave blank and fix manually). Enter comma-separated fix version(s) []: ^C ``` Only Ctrl-C exits, and the merge to `branch-4.2` has already been pushed by then, since the JIRA step runs in a `finally` block. After: ``` Target version for branch-4.2 is not found on JIRA, it may be archived or not created. Skipping it. Enter comma-separated fix version(s) []: No fix version entered; update SPARK-58880 manually. ``` In both cases the committer still has to record the Fix Version by hand. The point of the change is that the script now says so, instead of claiming the ticket is already correct or trapping them in a loop. ### Does this PR introduce _any_ user-facing change? No. Committer-facing only. ### How was this patch tested? New doctests on two extracted helpers, `fix_version_additions` and `fix_versions_from_input`, which cover both failure modes directly. `fix_versions_from_input("", "")` returns `[]` where the previous inline expression produced `[""]` and looped, and `fix_version_additions` separates "nothing inferred" from "all inferred already present", which the old code could not distinguish: ``` master : "".replace(" ","").split(",") -> [''] never a known version -> loops patched : fix_versions_from_input("", "") -> [] master : additional_fix_versions([], ["5.0.0"]) -> [] identical to the up-to-date case patched : fix_version_additions([], ["5.0.0"]) -> ([], False) patched : fix_version_additions(["5.0.0"], ["5.0.0"]) -> ([], True) ``` Doctests go from 80 to 88, and they run on every invocation of the script since `__main__` calls `doctest.testmod()` before `main()`. I also drove `resolve_jira_issue` with a stubbed JIRA client to produce the transcripts above and to confirm that the cases where a version is inferable are unchanged, including the resolved-ticket path that proposes additions and asks for confirmation. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor (Claude Opus 5) Closes apache#58300 from szehon-ho/SPARK-59017. Authored-by: Szehon Ho <szehon.apache@gmail.com> Signed-off-by: Szehon Ho <szehon.apache@gmail.com>

…eports a PartitioningCollection ### What changes were proposed in this pull request? Fix the post-shuffle regrouping in `EnsureRequirements` for storage-partitioned joins with `spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys=true`. The unwrap that pushes join key positions down to the keyed side only matches a bare `KeyedShuffleSpec`: ```scala bestSpecOpt match { case Some(KeyedShuffleSpec(_, _, Some(joinKeyPositions))) => withJoinKeyPositions(child, joinKeyPositions) case _ => child } ``` When the keyed child's `outputPartitioning` is a `PartitioningCollection` (e.g. an inner SPJ join result, or a projection duplicating a partition column under multiple aliases like `SELECT a AS a1, a AS a2, b`), its spec is a `ShuffleSpecCollection` wrapping the `KeyedShuffleSpec`, so the match falls through and no `GroupPartitionsExec` is inserted. This PR unwraps a (possibly nested) `ShuffleSpecCollection` to its head spec — the same spec `ShuffleSpecCollection.createPartitioning` delegates to — before the match. ### Why are the changes needed? Without the fix, the keyed side keeps its N ungrouped partitions while the other side is shuffled into the projected layout with M partitions, and planning fails with: ``` java.lang.IllegalArgumentException: requirement failed: All KeyedPartitionings in a PartitioningCollection must have equal partitionKeys ``` ### Does this PR introduce _any_ user-facing change? The affected queries used to fail with the error above; they now run correctly, shuffling only the non-keyed side. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 Closes apache#58315 from dongjoon-hyun/SPARK-59025. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…k serializers ### What changes were proposed in this pull request? This PR removes redundant boilerplate in `python/pyspark/serializers.py` that is either a Python 2 leftover or made unnecessary by Python 3 semantics. None of it changes behavior: - Drop `FramedSerializer.dumps`, a verbatim duplicate (signature and docstring) of the inherited abstract `Serializer.dumps`. - Drop `Serializer.__ne__`: Python 3 derives `!=` from `__eq__` automatically. `__hash__` is intentionally kept, since it is not auto-derived once `__eq__` is defined. - Drop the no-op `FramedSerializer.__init__(self)` call in `CompressedSerializer.__init__`: neither `Serializer` nor `FramedSerializer` defines `__init__`, so it only resolves to `object.__init__`. - Collapse the duplicate `itertools` import: remove `from itertools import chain, product` and qualify the four usages as `itertools.chain` / `itertools.product`, consistent with the existing `itertools.islice` usage in the same file. ### Why are the changes needed? These are dead or redundant lines that add noise without adding behavior. Removing them makes the module easier to read and maintain. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing `python/pyspark/tests/test_serializers.py` passes. The change is behavior-preserving: the removed `dumps` override and `__init__` call were no-ops, and `!=` continues to work via Python 3's automatic derivation from `__eq__`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#58297 from Yicong-Huang/cleanup-serializers-boilerplate. Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>

…teShuffleSpec` and `GroupPartitionsExec` ### What changes were proposed in this pull request? Extract the ordering of grouped partition keys into a shared helper, `KeyedPartitioning.groupedKeyRowOrdering`, and use it from both `KeyedPartitioning.keyRowOrdering` (used by `toGrouped`, and thus by `createShuffleSpec`'s subset-keys branch) and `GroupPartitionsExec.groupAndSortByKeys`. Add tests pinning that the two orders agree: a unit test comparing them directly, and a LEFT OUTER join test guarding the silent failure mode end to end. ### Why are the changes needed? With `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled`, one join side may be shuffled onto the key order declared by `createShuffleSpec`, while the keyed side's physical layout is produced by `GroupPartitionsExec`. The two sorts must be identical. If they diverged, inner joins would fail loudly at planning time (`ShuffledJoin` wraps both sides' partitionings into a `PartitioningCollection`, whose invariant requires equal partition keys), but join types that expose only one side's partitioning (e.g. LEFT OUTER) run nothing that compares the two orders and silently return wrong results -- verified by reversing the sort in `groupAndSortByKeys` and running a LEFT JOIN variant of the SPARK-58988 scenario, which loses join matches (`[1,aa,2021,null,null]` instead of `[1,aa,2021,1,aa]`). Today the two sorts agree only because both call `RowOrdering.createNaturalAscendingOrdering` on the same data types, and the requirement is recorded only in a comment. This fragility was noticed during the review of apache#58311 (SPARK-59022), which fixed a related but distinct hazard: an order that must follow another being derived independently. This PR is behavior-neutral and makes the contract shared, documented, and tested. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the newly added test coverage to ensure this contract. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 Closes apache#58318 from dongjoon-hyun/SPARK-59027. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…s deleted between get() calls ### What changes were proposed in this pull request? - Fetch the pod once in `ExecutorPodsLifecycleManager.removeExecutorFromK8s` instead of calling `podToDelete.get()` twice. ### Why are the changes needed? - `podToDelete` is a lazy `PodResource` handle, so each `get()` is a separate API server round trip. If the pod is deleted between the two calls, the second returns `null` and dereferencing `getMetadata` throws `NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Pod.getMetadata()" because the return value of "io.fabric8.kubernetes.client.dsl.PodResource.get()" is null`, thrown from `removeExecutorFromK8s` via `onFinalNonDeletedState` / `onNewSnapshots`. - Observed on a driver managing ~1000 executors during heavy pod churn. The exception aborts the K8s-side deletion for that executor, leaving the pod to be reaped on a later resync. - Introduced by SPARK-54197, which added the `deletionTimestamp` check as a second `get()` call rather than reusing the first result. The `&&` short-circuit guards against the first call returning `null`, but not against the pod disappearing between the two. ### Does this PR introduce _any_ user-facing change? - No. ### How was this patch tested? - New unit test in `ExecutorPodsLifecycleManagerSuite` stubbing `get()` to return the pod then `null` on successive calls. Fails on master with the NPE above, passes with this change. Full suite 10/10. ### Was this patch authored or co-authored using generative AI tooling? - Yes, using Claude with Opus 4.8. Closes apache#58291 from venkata91/SPARK-executor-pods-lifecycle-npe. Authored-by: Venkata krishnan Sowrirajan <venkat.sowrirajan@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…/VARCHAR ### What changes were proposed in this pull request? [SPARK-59001](https://issues.apache.org/jira/browse/SPARK-59001) (parent [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794)). Follow-up on first-class CHAR/VARCHAR (`spark.sql.charVarchar.standardSemantics.enabled`) so a few string-family call sites treat `CharType` / `VarcharType` like `STRING`. - `V1Writes` applies `Empty2Null` to nullable CHAR/VARCHAR partition columns (`dataType.isInstanceOf[StringType]`). `CHAR(n>0)` still pads `''` to spaces, so the empty CHAR case is `CHAR(0)`. - The V2 text data source accepts CHAR/VARCHAR as a string-family type. - Hive metastore filter conversion still refuses CHAR/VARCHAR partition keys (`varcharKeys` in `SupportedAttribute`: Hive's trailing-blank comparison is not Spark's). When a predicate mentions such a key, `prunePartitionsFastFallback` prunes client-side with Spark's own predicates (CHAR compared without PAD SPACE; the test literal is `'a '` for `CHAR(5)`). Other empty-filter and MetaException fallbacks still honor `metastorePartitionPruningFastFallback`. ### Why are the changes needed? With first-class types, CHAR/VARCHAR stay in the plan instead of being rewritten to annotated STRING. Equality against `StringType` then skips them: - empty partition values are not converted to NULL, so they become a distinct partition directory instead of `__HIVE_DEFAULT_PARTITION__` - `USING text` rejects a CHAR/VARCHAR schema - Hive partition filters on CHAR keys fetch every partition The annotation-skipping idea was tried and dropped: `ApplyCharTypePadding` uses `__CHAR_VARCHAR_TYPE_STRING` as an idempotence marker, so the annotation is load-bearing. ### Does this PR introduce _any_ user-facing change? Yes, only when `spark.sql.charVarchar.standardSemantics.enabled` is true (still off by default). - Empty `VARCHAR` / `CHAR(0)` partition values become NULL like STRING. - `spark.read.schema("value CHAR(n)").text(...)` is accepted. - Hive CHAR partition filters prune to the matching partitions (client-side), and `CHAR(5) = 'a'` does not match a stored `'a '` unless the literal carries the pad or an RTRIM collation is used. ### How was this patch tested? - `sql/testOnly *CharVarcharTestSuite *V1WriteCommandSuite`: 164 succeeded - `hive/testOnly *HiveCharVarcharTestSuite`: 58 succeeded - `hive/testOnly *HivePartitionFilteringSuite*`: 360 succeeded (Hive 2.3-4.1) ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 Closes apache#58255 from srielau/serge-rielau_data/SPARK-58794-p0-followup. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

### What changes were proposed in this pull request? This PR reverts apache#57389 (`bc9ee80d0de08f4cca1ac8518e4b6bee64cc544d`) on `master`. It restores the transient lazy leaf-to-index map used by `predictLeaf` and removes the post-construction leaf-index field, initialization passes, and associated persistence assertions. The resulting eight files exactly match their versions immediately before apache#57389. ### Why are the changes needed? apache#57389 replaces the per-leaf map by constructing an unindexed node graph and then rebuilding the entire tree to attach indices. Although trees in an ensemble are processed sequentially, this adds O(number of nodes in one tree) temporary driver memory during training, loading, and old-model conversion. A sufficiently large individual tree can therefore cause an out-of-memory failure. Reverting removes that regression while an implementation that assigns immutable indices during initial node construction is considered separately in apache#58330. ### Does this PR introduce _any_ user-facing change? No compared with a released Spark version. This restores the behavior that existed before the unreleased change in apache#57389. Leaf IDs and their traversal order are unchanged. ### How was this patch tested? The revert was verified to apply cleanly to current `master`, and all affected files were compared with their pre-apache#57389 versions. Targeted ML tests and Scala lint have not been run yet; the PR was opened first as requested. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) Closes apache#58332 from zhengruifeng/ml-revert-leaf-index-dev-7. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>

### What changes were proposed in this pull request? When a conflict occurs on cherry picking a commit to a branch, ask the user to fix the conflict but have the script continue the cherry pick itself. ### Why are the changes needed? If the user calls `git cherry-pick --continue` themselves, commit message lines starting with `#` will be interpreted as comments and stripped. This is [what happened to me here][1]. Compare the commit message on apache@9a75a1d (master) to apache@8cc56ea (branch-4.x) and note how every line beginning with `#` was stripped from the latter. [1]: apache#58136 (comment) I'm not the only one who has hit this. It seems this has been happening for a while. #### Cherry-Pick Commit Message Loss Cases In all of these cases, the `branch-4x` commit messages are malformed relative to the originals from `master`. | Date | `master` source | `branch-4.x` backport | |---|---|---| | 2026-07-21 | [99025ce](apache@99025ce) | [98e98b7](apache@98e98b7) | | 2026-07-20 | [6b719b3](apache@6b719b3) | [6c0a252](apache@6c0a252) | | 2026-07-14 | [226340c](apache@226340c) | [67df419](apache@67df419) | | 2026-07-11 | [710b3c4](apache@710b3c4) | [da117f6](apache@da117f6) | | 2026-06-18 | [880083f](apache@880083f) | [84fcfba](apache@84fcfba) | | 2026-06-16 | [2fb4a1b](apache@2fb4a1b) | [07d9f34](apache@07d9f34) | | 2026-06-03 | [13b526d](apache@13b526d) | [7a70689](apache@7a70689) | | 2026-05-25 | [0af3d42](apache@0af3d42) | [bdd5fdf](apache@bdd5fdf) | | 2026-05-16 | [0a0d31b](apache@0a0d31b) | [f5273c7](apache@f5273c7) | | 2026-05-13 | [436291e](apache@436291e) | [f67a855](apache@f67a855) | | 2026-05-08 | [bb72aef](apache@bb72aef) | [e7ae20a](apache@e7ae20a) | ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I used this test script to confirm that `commit.cleanup=scissors` preserves Markdown headings and other lines that begin with `#` when there is a cherry pick conflict. [test-cherry-pick-commit-msg.sh](https://github.com/user-attachments/files/31342347/test-cherry-pick-commit-msg.sh) I also tried the merge script from this branch with `--dry-run` and it worked, including a conflicted cherry pick to `branch-4.x`. ### Was this patch authored or co-authored using generative AI tooling? I wrote this with assistance from GitHub Copilot. Closes apache#58214 from nchammas/merge-pr-cherry-pick-whitespace. Authored-by: Nicholas Chammas <nicholas.chammas@gmail.com> Signed-off-by: Nicholas Chammas <nicholas.chammas@gmail.com>

…ming metrics reporting ### What changes were proposed in this pull request? This PR fixes a `NullPointerException` in `KafkaMicroBatchStream.metrics()` during Kafka micro-batch streaming progress reporting (`finishTrigger`). Specifically, `KafkaMicroBatchStream.metrics()` checked `latestAvailablePartitionOffsets.isDefined` before extracting partition offsets. If `latestAvailablePartitionOffsets` was `Some(null)`, `isDefined` returned `true`, causing `latestAvailablePartitionOffsets.get` to return `null` and throwing a `NullPointerException` when `.map()` was invoked on it. This PR updates the condition to `latestAvailablePartitionOffsets.exists(_ != null)` to safely ensure partition offsets are non-null before invoking `.map()`. ### Why are the changes needed? When uninitialized partition offsets or race conditions occur during progress reporting (`finishTrigger`), `latestAvailablePartitionOffsets` can be `Some(null)`. Without this check, calling `.map()` on `null` throws a `NullPointerException`, crashing the entire streaming query job in production after batch execution. ### Does this PR introduce _any_ user-facing change? No API changes. Fixes a `NullPointerException` in progress reporting metrics. ### How was this patch tested? Added unit test assertion in `KafkaMicroBatchSourceSuite`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#58133 from zahed1994/SPARK-55271-kafka-streaming-metrics-npe. Authored-by: zahed1994 <zahedshareef@gmail.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>

### What changes were proposed in this pull request? This is a test-only PR for SPARK-58641. It adds SQL coverage and regenerated golden files without changing production code. Expand the `identifier-clause` SQL tests to cover identifier names computed through standard function resolution, including built-in and higher-order functions, aggregate/window/generator expressions, SQL functions and their dependencies, relation identifiers, and scalar subqueries. Regenerate the standard and legacy analyzer and execution golden files. ### Why are the changes needed? These cases provide coverage for function resolution while computing `IDENTIFIER` names and keep the shared identifier golden files synchronized across Spark analyzer implementations. ### Does this PR introduce _any_ user-facing change? No. This is a test-only change. ### How was this patch tested? - Regenerated the standard and legacy analyzer and execution golden files. - Ran `./build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z identifier-clause"`. - All 4 matching tests passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) Closes apache#57850 from vladanvasi-db/effort/identifier-spark. Authored-by: Vladan Vasić <vladan.vasic@databricks.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

…nt part file rename ### What changes were proposed in this pull request? `ReliableCheckpointRDD.writePartitionToCheckpointFile` renames the attempt-temp file onto the final part file and only handles a rename that reports failure by returning `false` (HDFS semantics). This PR additionally catches `FileAlreadyExistsException` from that rename and routes it into the same existing handling: if the final part file exists, another attempt of this task already committed it, so the temp file is deleted and the write is treated as successful. If the destination does not exist, the existing `checkpointFailedToSaveError` is still thrown. ### Why are the changes needed? Since [HADOOP-16721](https://issues.apache.org/jira/browse/HADOOP-16721) (Hadoop 3.3.1), S3A deliberately raises `FileAlreadyExistsException` when the rename destination is an existing file, instead of returning `false` as HDFS does. ABFS behaves the same way. The Hadoop FileSystem specification does not guarantee HDFS-style `false` reporting. Under speculative execution (or a zombie attempt racing a retry), two attempts of the same checkpoint task race to rename onto the same final part file. On HDFS the loser sees `rename() == false` and Spark correctly treats it as "some other copy of this task must've finished before us". On S3A/ABFS the loser gets an unhandled `FileAlreadyExistsException`, which fails the task — and because the destination now permanently exists, **every retry of that task fails on the same rename**, so `spark.task.maxFailures` is always exhausted and the job aborts, even though the checkpoint data was written successfully by the winning attempt. Observed in production (Spark 4.0.1, Hadoop 3.4.1, `spark.speculation=true`): ``` org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename s3://<bucket>/<prefix>/spark-checkpoints/<uuid>/rdd-207/.part-00379-attempt-25364 to s3://<bucket>/<prefix>/spark-checkpoints/<uuid>/rdd-207/part-00379; destination file exists at org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2468) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2392) at org.apache.spark.rdd.ReliableCheckpointRDD$.writePartitionToCheckpointFile(ReliableCheckpointRDD.scala:229) ... ERROR TaskSetManager: Task 379 in stage 105.0 failed 4 times; aborting job ``` See [SPARK-58750](https://issues.apache.org/jira/browse/SPARK-58750) for full details. Structured Streaming's `CheckpointFileManager` was already hardened for divergent rename semantics; the RDD checkpoint writer is the remaining caller assuming HDFS semantics. ### Does this PR introduce _any_ user-facing change? No. RDD checkpointing to S3A/ABFS under speculative execution (or task retry after a committed rename) now succeeds instead of unrecoverably failing the job, which is the bug fix itself. ### How was this patch tested? Added a regression test to `CheckpointStorageSuite` that writes the same checkpoint partition from two task attempts against a `FileSystem` mimicking S3A's rename semantics (raises `FileAlreadyExistsException` when the destination file exists). Without the fix, the second attempt throws and would fail the task; with the fix, it succeeds and exactly one committed part file remains, with the attempt-temp file cleaned up. Ran `build/sbt "core/testOnly org.apache.spark.CheckpointStorageSuite"` locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (model claude-fable-5) Closes apache#57976 from james-willis/SPARK-58750. Authored-by: James Willis <james@wherobots.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…rage and reject nested filter attributes ### What changes were proposed in this pull request? `SupportsRuntimeCatalystFiltering` was added by SPARK-58523, which covered row-level operations (`RowLevelOperationCatalystRuntimeFilterSuiteBase` and its group- and delta-based subclasses) and a core set of pushdown tests. It was not exercised in the two other places the predicate-based runtime filtering interfaces are: dynamic partition pruning and storage-partitioned joins. This PR closes that gap by reusing the existing suites rather than cloning them, and hardens the `filterAttributes()` contract. **Behavior change** - `DataSourceV2ScanRelation.runtimeFilterAttrs` (and `fullyPushedRuntimeFilterAttrs`) now reject a nested (multi-part) runtime-filter reference, including one over a struct column. Both runtime filtering interfaces require each `filterAttributes()` reference to be a top-level read-schema attribute; a nested reference such as `s.tz` previously resolved and widened to the enclosing struct column `s`, making runtime filters over every field of `s` eligible. **Test fixtures** - `InMemoryBaseTable`: `CatalystRuntimeFilteringScan` now prunes on nested partition keys. It previously looked up a partition attribute by joining `fieldNames` into a single top-level name, so a `GetStructField` chain such as `s.part` never matched and no pruning happened. Partition attributes now keep their `fieldNames` as name-part sequences, and `partitionAttrFor` matches the expression's path component-wise, so a quoted top-level column `` `a.b` `` stays distinct from a nested `a`.`b`. - `InMemoryCatalystRuntimeFilterTable`: threads the full table-creation metadata (constraints, distribution, ordering, partition counts, advisory size, strictness, and `numRowsPerSplit`) through to `InMemoryBaseTable`, and derives `filterAttributes()` / `fullyPushedFilterAttributes()` from a shared helper. - `InMemoryTableCatalystRuntimeFilterCatalog`: adds `InMemoryCatalystRuntimeFilterCatalog`, the `InMemoryCatalog` counterpart, so the Catalyst fixture can be used where functions and procedures are needed. - `InMemoryTableWithV2Filter`: threads the same table-creation metadata through to `InMemoryBaseTable`. **Reused suites** - `DynamicPartitionPruningSuite`: adds `DynamicPartitionPruningV2CatalystFilterSuiteAEOff` / `AEOn`, mirroring the existing `DynamicPartitionPruningV2FilterSuite` pair. - `KeyGroupedPartitioningSuite`: the shared SPJ fixtures move to a new `KeyGroupedPartitioningSuiteBase`, and the three runtime-filtering tests move to a `KeyGroupedPartitioningRuntimeFilterTests` trait. Two small suites then run those three tests once per interface: `KeyGroupedPartitioningRuntimeFilterSuite` (predicate-based, the default `InMemoryCatalog`) and `KeyGroupedPartitioningCatalystRuntimeFilterSuite`. - `DistributionAndOrderingSuiteBase`: `catalogClassName` becomes overridable so a subclass can vary the catalog behind `testcat`. Note for reviewers: `SPARK-42038: partially clustered: with dynamic partition filtering` and `SPARK-45652: SPJ should handle empty partition after dynamic filtering` are unchanged, but now report under `KeyGroupedPartitioningRuntimeFilterSuite` instead of `KeyGroupedPartitioningSuite`. Most of the diff in that file is this movement. **New tests in `DataSourceV2CatalystRuntimeFilterSuite`** Covering behavior specific to pushing Catalyst expressions: a DPP filter on a nested partition field arriving with the nested access intact, multiple predicates pushed in a single `filter()` call, a filter with no V2 translation being pushed instead of dropped, a scan implementing both runtime filtering interfaces being rejected, two partition columns whose dotted names collide (a quoted top-level `` `x.y` `` and a nested `x`.`y`) binding to the correct partition slot, and the `filterAttributes()` contract when a reported attribute is not a top-level scan attribute -- a missing attribute and a nested reference (over both an int and a struct column) are all rejected. One test calls `PushDownUtils.replanWithRuntimeFilters` directly, to reach the SPJ partitioning-preservation checks that a well-behaved source cannot trigger: dropping `HasPartitionKey`, reporting a partition key that was not in the original partitioning, or growing a key's split count. ### Why are the changes needed? `SupportsRuntimeCatalystFiltering` is the path a scan takes when runtime filters are pushed as Catalyst expressions instead of connector predicates. Dynamic partition pruning and storage-partitioned joins are the two features that produce those filters, and neither was tested against this interface, so regressions in the Catalyst path would not have been caught by the suites that cover the equivalent predicate-based path. ### Does this PR introduce _any_ user-facing change? No end-user-facing behavior change. For connector authors, a scan that declares a nested runtime-filter attribute -- a violation of the `filterAttributes()` contract -- is now rejected with an internal error instead of silently widening to the enclosing top-level column. ### How was this patch tested? Existing and new unit tests. Locally, on the rebased branch: - `DataSourceV2CatalystRuntimeFilterSuite`: 16 tests passed, including the new dotted/nested collision test and the nested-reference rejection. - `DynamicPartitionPruningV2CatalystFilterSuiteAEOff` / `AEOn` and `DynamicPartitionPruningV2FilterSuiteAEOff` / `AEOn`: 154 tests passed. - `KeyGroupedPartitioningSuite`: 96 tests passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor with Claude Opus 5 Closes apache#58296 from szehon-ho/catalyst-runtime-filter-coverage. Authored-by: Szehon Ho <szehon.apache@gmail.com> Signed-off-by: Szehon Ho <szehon.apache@gmail.com>

### What changes were proposed in this pull request? This PR lets a driver report its hold status to the standalone Master, so that the Master UI and `/json/` endpoint can show it. Display only -- the `(hold)` / `(resume)` controls stay on the driver web UI. - `SparkContext` calls a new no-op `CoarseGrainedSchedulerBackend.reportExecutorHoldStatus` hook after initialization and on every hold/resume transition. `StandaloneSchedulerBackend` overrides it and forwards to `StandaloneAppClient`, which sends the new `ApplicationHoldUpdated` message to the Master. The status is cached and re-sent on registration and failover. - The Master mirrors it onto `ApplicationInfo.holdSupported` / `held` (`transient`; a new Master learns it again from the driver's re-report). The draining count is not pushed: the Master derives it from the executors it already tracks. - The Master UI annotates the state column (e.g. `RUNNING (held, draining 2 executors)`), and `/json/` gains `holdsupported`, `held`, and `draining` fields per application. `spark.ui.holdEnabled` is honored (an opted-out application is not reported as holdable), and finished applications are never annotated. No new configuration and no new public API. ### Why are the changes needed? SPARK-58828 and SPARK-59010 made the hold status visible only per application on the driver. An operator on the Master page cannot tell which applications are held or still draining. This also lays the groundwork for offering the controls on the Master UI in a follow-up PR. ### Does this PR introduce _any_ user-facing change? Yes, additive: the Master UI annotates held applications in the state column, and each application in `/json/` gains `holdsupported`, `held`, and `draining` fields. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 Closes apache#58346 from dongjoon-hyun/SPARK-59055. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…k context ### What changes were proposed in this pull request? Adds the `cpuAmount` key to the mock task-context JSON that `MockProtocolWriter` writes in `python/benchmarks/bench_eval_type.py`. ### Why are the changes needed? Every benchmark in `bench_eval_type.py` currently fails at worker initialization. `TaskContextInfo.from_stream` reads `task_context_json["cpuAmount"]` without a default (`python/pyspark/worker_message.py`), so the worker raises `KeyError: 'cpuAmount'` before any UDF runs: ``` File "python/pyspark/worker_message.py", line 63, in from_stream cpu_amount=Decimal(task_context_json["cpuAmount"]), KeyError: 'cpuAmount' ``` The key became required in SPARK-58192, which added it on the JVM side and in `worker_message.py` but did not update this mock. The failure is easy to miss because `setup()` only builds the input buffer; only running `time_worker` / `peakmem_worker` reaches the worker. Nothing under `python/benchmarks/` is listed in `dev/sparktestsupport/modules.py`, so no CI job exercises these benchmarks. ### Does this PR introduce _any_ user-facing change? No. Benchmark-only change. ### How was this patch tested? Ran every benchmark class in the file, driving the worker end to end: | | before | after | |---|---|---| | `*TimeBench` | 0 of 26 | 26 of 26 | | `*PeakmemBench` | 0 of 26 | 26 of 26 | Also checked out each release branch and ran the same benchmarks there: | branch | result | |---|---| | `master` | 0 of 26; 26 of 26 with this fix | | `branch-4.x` | same failure; the same one-line fix applies | | `branch-4.3` | 0 of 26; 26 of 26 with this fix | | `branch-4.2` | unaffected -- no `worker_message.py`, so the key is not required | So this is worth backporting to `branch-4.x` and `branch-4.3`. The value is a plain decimal string, matching `CpuAmount.toDisplayString` on the JVM side (`core/src/main/scala/org/apache/spark/api/python/PythonWorkerUtils.scala`), and is consistent with the existing `"cpus": 1` in the same mock. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) Closes apache#58353 from viirya/SPARK-bench-cpuamount. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>

### What changes were proposed in this pull request? Given the same CDF (but can be shuffled in arbitrary order), every key's output row in SCD1 should be exactly equal (in data columns, not necessarily CDC metadata) to the last row per key in SCD2. In other words, SCD1 and SCD2 should converge. Using this principle and reusing the existing AutoCDC out-of-order convergence test harness, we should setup a cross-SCD convergence test so that one implementation can keep the other in check for correctness, and detect regressions made to one SCD's implementation but not the other. Concretely, two things are accomplished in this PR: 1. Extract shared convergence-testing infra from `AutoCdcOutOfOrderConvergenceSuite` into a new `AutoCdcRandomCdcTestMixin` trait 2. Implement the cross-SCD convergence test in `AutoCdcCrossScdConvergenceSuite` ### Why are the changes needed? Correctness and regression testing of AutoCDC SCD1 and SCD2 implementations. ### Does this PR introduce _any_ user-facing change? No, test-only change. ### How was this patch tested? Added `AutoCdcCrossScdConvergenceSuite`. Test-only change. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Claude Sonnet 5 and GPT-5.6 Sol. Closes apache#58055 from AnishMahto/SPARK-58572-cross-scd-fuzz-testing. Authored-by: AnishMahto <anish.mahto99@gmail.com> Signed-off-by: Szehon Ho <szehon.apache@gmail.com>

### What changes were proposed in this pull request? This is layer 4 of the nine-PR local Connect pool stack: apache#57684 -> apache#57685 -> apache#57907 -> apache#57686 -> apache#58247 -> apache#57687 -> apache#58248 -> apache#57102 -> apache#57688 The three lower layers are merged, so GitHub shows only this layer's two-file diff. The review unit introduced here is commit `37dbe4b5ae7`. This layer adds the normal server-retirement path on top of member claiming: - atomic JSON state replacement and cleanup of interrupted-write temporary files; - validated retired-state records with recovery of independently valid process IDs; - crash-safe state-to-retired renames; - graceful SIGTERM shutdown followed by bounded SIGKILL escalation; and - idempotent claimed-member release, including forked-child and retry handling. Janitor recovery, acquisition, forceful purge, SparkSession integration, and warmup remain in later PRs. ### Why are the changes needed? A claimed server must be released without losing the only process handle if the client crashes during the state transition. Isolating retirement keeps atomic persistence, graceful shutdown, and release semantics independently reviewable before orphan scanning and launch orchestration are added. ### Does this PR introduce _any_ user-facing change? No. The pool is not wired into SparkSession in this layer. ### How was this patch tested? Added eight focused tests at this layer, bringing the suite to 33 tests, and extended the storage test with interrupted-write and stale-temporary-file coverage. The focused command is: ```bash python/run-tests --testnames pyspark.sql.tests.connect.test_connect_local_server_pool ``` At the stack tip, the equivalent direct `unittest` invocation passed all 56 pool tests, including the two real-server E2E tests. The rebuilt commit passed Python AST parsing, `git diff --check`, and changed-file ASCII and 100-column checks. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Fable 5) and OpenAI Codex (GPT-5) Closes apache#57686 from ericm-db/local-connect-pool-lifecycle. Authored-by: Eric Marnadi <eric.marnadi@databricks.com> Signed-off-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>

…g tests ### What changes were proposed in this pull request? Wrap the log read-and-assert step of `ApplyInArrowTests.test_apply_in_arrow_with_logging` and `test_apply_in_arrow_iter_with_logging` in `eventually` so it polls until the asynchronously captured worker logs are visible. The `applyInArrow` call that produces the logs stays outside the poll so it runs once. ### Why are the changes needed? Observed on fork CI: the `pyspark-sql` job failed with `[DIFFERENT_ROWS]` (100%) where `spark.tvf.python_worker_logs()` returned no rows instead of the expected two WARNING rows, then re-ran green on the next attempt: https://github.com/Yicong-Huang/spark/actions/runs/32828898783/job/97748219772 Root cause: worker logs are captured asynchronously. Python workers emit log records on stdout; on the JVM side a per-worker `RedirectThread` (`PythonWorkerLogCapture`) drains stdout and only saves the log block to the `BlockManager` once it reads the trailing marker line. That drain runs independently of the query result, which returns over a separate socket channel, so `python_worker_logs()` invoked right after the query can observe zero blocks. Polling the read side waits out the race. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Built with `build/sbt -Phive package` and ran `python/run-tests --testnames 'pyspark.sql.tests.arrow.test_arrow_grouped_map'`; both logging tests and the full module pass. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#58289 from Yicong-Huang/flaky-worker-logs. Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>

Co-authored-by: Isaac Claude-Session: https://claude.ai/code/session_011tpdLYqY4VRw4s1KjDb2fi

github-actions Bot added BUILD SQL PYTHON CONNECT labels Jul 8, 2026

ericm-db force-pushed the local-connect-reuse branch from f4d9735 to 5f1f97b Compare July 30, 2026 01:01

ericm-db force-pushed the local-connect-pool branch 3 times, most recently from f43947b to 320bffc Compare July 31, 2026 18:20

ericm-db force-pushed the local-connect-pool branch from 320bffc to 0c53b32 Compare August 10, 2026 20:15

Spenserrrr and others added 20 commits August 13, 2026 15:35

zhengruifeng and others added 29 commits August 26, 2026 16:25

[SPARK-58021][CONNECT] Reap orphaned local pool members

0fe123a

[SPARK-58021][CONNECT] Track local pool claimant process generations

3de7b10

[SPARK-58021][CONNECT] Recover abandoned local pool launches

a5dc822

[SPARK-58021][CONNECT] Add local server pool acquisition engine

6434e57

[SPARK-58021][CONNECT] Add forceful local pool purge

50b735d

[SPARK-58021][CONNECT] Integrate local server pools with SparkSession

6aa073c

Co-authored-by: Isaac Claude-Session: https://claude.ai/code/session_011tpdLYqY4VRw4s1KjDb2fi

ericm-db force-pushed the local-connect-pool branch from 832dacc to 6aa073c Compare August 28, 2026 00:36

Navigation Menu

Connect Pool - #30

Connect Pool#30
ericm-db wants to merge 520 commits into
local-connect-reusefrom
local-connect-pool

ericm-db commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

ericm-db commented Jul 8, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jul 8, 2026

⚠️ Pull Request Title Validation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants