Skip to content

Vertex AI default environment has broken cupy installation #709

Description

@ncclementi

In the altest Vertex AI env, the cuda toolkit version is 13

ls -ld /usr/local/cuda*
lrwxrwxrwx  1 root root   21 Jun  1 04:54 /usr/local/cuda -> /usr/local/cuda-13.0/
drwxr-xr-x 17 root root 4096 Jun  1 04:55 /usr/local/cuda-13.0

but when checking pip list they have a cuda 12 installation

cupy-cuda12x                          13.6.0

Which is a broken env:

import cupy as cp
X = cp.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1],[3, 2]], cp.float32)
X**2
RuntimeError: CuPy failed to load libnvrtc.so.12: OSError: libnvrtc.so.12: cannot open shared object file: No such file or directory

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions