Skip to content

Revert torch version to 2.5.1 - #157

Open
hamadfadi wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
hamadfadi:fix-torch-version
Open

Revert torch version to 2.5.1#157
hamadfadi wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
hamadfadi:fix-torch-version

Conversation

@hamadfadi

Copy link
Copy Markdown
Collaborator

Restores torch==2.5.1+cu121 and its corresponding wheel hashes in user_examples/llm_fine_tuning_cloud_batch/requirements.txt.

### Motivation & Root Cause Analysis

Attempting to upgrade to `torch==2.13.0` introduced critical runtime and build failures on the CUDA 12 infrastructure stack:

1. **C-Symbol Linkage Mismatch (`ncclCommResume`)**:
   - The platform base container (`infrastructure/requirements.txt`) installs CUDA 12 drivers and `nvidia-nccl-cu12` (v2.21.5).
   - `torch 2.13.0` requires CUDA 13 and `nvidia-nccl-cu13` (NCCL 2.29+), which calls the new `ncclCommResume` C-symbol.
   - At runtime, Python's dynamic loader loads `libnccl.so.2` from `nvidia-nccl-cu12` first. When `libtorch_cuda.so` attempts GPU initialization, it fails with:
     ```text
     /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so: undefined symbol: ncclCommResume
     ```
   - This causes GPU training loops to fail and silently fall back to surrogate mode.

2. **PyPI `--require-hashes` Version Lock Conflict**:
   - PyPI's official CUDA 13 wheels have mutually exclusive internal pins (`cuda-toolkit 13.0.3` pins `nvidia-cublas==13.1.1.3.*` while `nvidia-cudnn-cu13` pins `nvidia-cublas==13.6.1.10`). Under strict `--require-hashes` mode, `pip` cannot resolve the dependency graph.

### Solution
Restoring `torch==2.5.1+cu121` aligns PyTorch with the CUDA 12 base libraries (`nvidia-nccl-cu12`), eliminating C-symbol mismatches and ensuring real GPU fine-tuning executes properly on Cloud Batch worker VMs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants