tensorstore: add build-tensorstore.yml for riscv64 wheels - #1865
Merged
Merged
Conversation
luhenry
added a commit
that referenced
this pull request
Sep 12, 2026
…eeded numpy/ml_dtypes at build time
…cm version git apply leaves the tensorstore checkout dirty, which setuptools_scm reads as a post-release dev build; pin it explicitly with SETUPTOOLS_SCM_PRETEND_VERSION_FOR_TENSORSTORE instead.
…red riscv64 runner pool The previous run's self-hosted runner lost communication with the server partway through the build step (GitHub annotation: CPU/memory starvation), while three other packages' builds were running concurrently on the same runner pool. --local_ram_resources=HOST_RAM*.5 throttles bazel's own action scheduling so it doesn't assume the whole host is available to it.
…n riscv64 Bazel's own pip-install repository rule has no riscv64 wheel for the hash-pinned numpy==2.4.6 anywhere, so it falls back to a from-source build that blows past its 600s default timeout. PIP_ONLY_BINARY=numpy plus the patch keep it on our registry's prebuilt wheel instead.
…m64/x86_64 adler32.c, crc32.c and deflate.c unconditionally #include cpu_features.h, but the vendored zlib.BUILD.bazel only added it to srcs for arm64/x86_64, so any other target cpu (riscv64 included) fails with a missing-header compile error.
… docs/ The single job looped cp311..cp314t inside one bazel output base, expecting the C++ world to be built once and reused. It is not: every interpreter reconfigures the whole build, so each wheel cost ~5h30 and the fifth one was still running when the 1440-minute timeout killed the job with four finished wheels on disk. A matrix over the interpreters keeps each job at ~5h30, well inside the 720-minute timeout the other bazel ports use. Also brings the workflow to the current conventions: a push trigger, the version taken from docs/packages/tensorstore.yaml through _setup.yml instead of a workflow input, the registry's default cp312-cp314t matrix, and auditwheel repair in place of renaming the wheel file, so the platform tag inside the wheel matches its name.
luhenry
force-pushed
the
tensorstore
branch
from
September 20, 2026 21:27
7a2887d to
cf29c7b
Compare
Contributor
|
luhenry
added a commit
that referenced
this pull request
Sep 20, 2026
…he other bazel ports Bootstrapping 8.5.1 from the dist archive took 174 minutes on a riscv64 runner, against a 180-minute timeout; the shared cache expires after a week of no hits, so the next cold run would decide it on runner load.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tensorstore0.1.85Bazel-built C++ library for reading/writing large multi-dimensional arrays, exposed as a Python extension. Upstream publishes no riscv64 wheel.
Mirrors upstream's
build.yml, driving the container directly instead of thecibuildwheelCLI so a bootstrappedbazelbinary stays under our control.Differs from upstream
bazel-8.5.1-dist.zipdist archive - bazelisk publishes no riscv64 binary. Its own job, cached repo-wide on the bazel version; a cold bootstrap takes ~175 min on a riscv64 runner.TENSORSTORE_BAZELISKpoints at a thin exec shim around that binary instead of bazelisk.py.docker runstep instead of thecibuildwheelCLI, one job per interpreter. A single job looping the interpreters was tried first and does not work here:setup.pyreconfigures the whole bazel build per interpreter, so each wheel costs a full ~5h30 build with nothing shared, and the job overran a 24h timeout with four of five wheels done.SETUPTOOLS_SCM_PRETEND_VERSION_FOR_TENSORSTOREpins the version - applying the patches below leaves the checkout dirty, which setuptools_scm otherwise reads as a post-release dev build.--local_ram_resources=HOST_RAM*.5keeps bazel from oversubscribing memory on the shared riscv64 runner pool, where the four legs now run concurrently.auditwheel repair --plat manylinux_2_39_riscv64produces the final wheel, as inbuild-grain.ymlandbuild-array-record.yml.Matrix: cp312, cp313, cp314, cp314t - the registry's default. Upstream also builds cp311 (
requires-python >= 3.11); numpy's 3.12 floor is what the registry follows.Testing
python_test_requirements_frozen.txtpins (numpy 2.4.6 / ml-dtypes 0.5.4 have no riscv64 wheel on our registry yet).License: OK
Patches
0001-Add-riscv64-support-to-the-dav1d-Bazel-BUILD-wrapper.patch- To upstream (google/tensorstore). Vendored dav1d BUILD fileselect()s only branch on arm64/ppc/x86_64, failing bazel analysis on riscv64. Reproduces riscv64-only.0002-Add-a-riscv64-default-to-libaom-s-header-select.patch- To upstream (google/tensorstore). Same gap in vendored libaom's headerselect(), pulled in transitively via the avif image driver. Reproduces riscv64-only.0003-Unpin-the-hermetic-numpy-fetch-used-for-building-hea.patch- Inappropriate. Bazel's own pip-install repo rule hash-pins a numpy version with no riscv64 wheel anywhere, so it times out building it from source; drops the pin so it resolves to our registry's wheel. Reproduces riscv64-only.0004-Ship-zlib-s-cpu_features.c-.h-on-every-arch-not-just.patch- To upstream (google/tensorstore). Vendored zlib BUILD file only ships cpu_features.c/.h on arm64/x86_64, but three base zlib sources include it unconditionally; affects any non-arm64/x86_64 target, riscv64 included.