[Question]: One of pod nvidia-cuda-validator-rr9qz 0/1 Init:Error

[AleksFirsta]

Hello Team

In my cluster I cant fix pod one of the pod

gpu-feature-discovery-9ztn5                                       1/1     Running      0               16m
gpu-feature-discovery-nlg6b                                       1/1     Running      0               16m
gpu-feature-discovery-rmvtr                                       1/1     Running      0               16m
gpu-operator-75b97bdcbc-fjmsm                                     1/1     Running      0               17m
nvidia-container-toolkit-daemonset-65m2r                          1/1     Running      0               16m
nvidia-container-toolkit-daemonset-prg6k                          1/1     Running      0               16m
nvidia-container-toolkit-daemonset-s9g8m                          1/1     Running      0               16m
nvidia-cuda-validator-44rfx                                       0/1     Completed    0               14m
nvidia-cuda-validator-7bh8j                                       0/1     Completed    0               13m
nvidia-cuda-validator-rr9qz                                       0/1     Init:Error   4 (2m14s ago)   5m1s
nvidia-dcgm-exporter-bn65s                                        1/1     Running      0               16m
nvidia-dcgm-exporter-f454z                                        1/1     Running      0               16m
nvidia-dcgm-exporter-z8n4w                                        1/1     Running      0               16m
nvidia-device-plugin-daemonset-k4rl8                              1/1     Running      0               16m
nvidia-device-plugin-daemonset-p7w7s                              1/1     Running      0               16m
nvidia-device-plugin-daemonset-vt5d7                              1/1     Running      0               16m
nvidia-driver-daemonset-9hcbr                                     1/1     Running      0               17m
nvidia-driver-daemonset-dhq4j                                     1/1     Running      0               16m
nvidia-driver-daemonset-t84wg                                     1/1     Running      0               16m
nvidia-gpu-operator-node-feature-discovery-gc-796ccf46c6-p9674    1/1     Running      0               17m
nvidia-gpu-operator-node-feature-discovery-master-65589f87tw7qq   1/1     Running      0               17m
nvidia-gpu-operator-node-feature-discovery-worker-62nv7           1/1     Running      1 (16m ago)     17m
nvidia-gpu-operator-node-feature-discovery-worker-gjvjg           1/1     Running      0               17m
nvidia-gpu-operator-node-feature-discovery-worker-jvmqm           1/1     Running      1 (16m ago)     17m
nvidia-gpu-operator-node-feature-discovery-worker-pxkgq           1/1     Running      0               17m
nvidia-gpu-operator-node-feature-discovery-worker-qgrbd           1/1     Running      1 (16m ago)     17m
nvidia-gpu-operator-node-feature-discovery-worker-t9rq6           1/1     Running      1 (16m ago)     17m
nvidia-mig-manager-jq47r                                          1/1     Running      0               15m
nvidia-mig-manager-l6wdd                                          1/1     Running      0               13m
nvidia-mig-manager-twltl                                          1/1     Running      0               14m
nvidia-operator-validator-hpqxf                                   1/1     Running      0               16m
nvidia-operator-validator-tc7b2                                   1/1     Running      0               16m
nvidia-operator-validator-zfrz6                                   0/1     Init:2/4     2 (5m13s ago)   16m

I did node restarts, daemonSet restart but nothing can helped me

GPU Operator version: v26.3.2
Driver: 580.159.04
GPU: NVIDIA B200
OS Type Ubuntu 24.04
CUDA Version: 13.0

[rahulait]

Hi @AleksFirsta , can you check the output of nvidia-smi -q and look for fabric state? Based on the logs you have provided, I see it still "in progress" and that is blocking further install.

    Fabric
        State                                          : In Progress
        Status                                         : N/A
        CliqueId                                       : N/A
        ClusterUUID                                    : N/A
        Health
            Summary                                    : N/A
            Bandwidth                                  : N/A
            Route Recovery in progress                 : N/A
            Route Unhealthy                            : N/A
            Access Timeout Recovery                    : N/A
            Incorrect Configuration                    : N/A
            Partition Assigned                         : N/A

Can you check what it is on other nodes where everything is working fine.

[AleksFirsta]

Hello @rahulait
Thanks for response. While I waited response - I did reinstall for all 3 workers. Case didnt changed.

Bellow I provide you required information:

gpu-feature-discovery-28w8z                                       1/1     Running                 0                3h39m
gpu-feature-discovery-8d7sm                                       1/1     Running                 0                3h35m
gpu-feature-discovery-jlj9h                                       1/1     Running                 0                169m
gpu-operator-1780411155-node-feature-discovery-gc-68f5b5c7cn8m6   1/1     Running                 0                3h47m
gpu-operator-1780411155-node-feature-discovery-master-86f5x2g5b   1/1     Running                 0                4h4m
gpu-operator-1780411155-node-feature-discovery-worker-4dqlv       1/1     Running                 1 (4h3m ago)     4h4m
gpu-operator-1780411155-node-feature-discovery-worker-78kjd       1/1     Running                 3 (4h3m ago)     4h4m
gpu-operator-1780411155-node-feature-discovery-worker-fj422       1/1     Running                 1 (4h3m ago)     4h4m
gpu-operator-1780411155-node-feature-discovery-worker-gtxgs       1/1     Running                 1 (4h4m ago)     4h4m
gpu-operator-1780411155-node-feature-discovery-worker-m92bw       1/1     Running                 1 (4h4m ago)     4h4m
gpu-operator-1780411155-node-feature-discovery-worker-nj9qh       1/1     Running                 0                4h4m
gpu-operator-5fc965585d-54ttx                                     1/1     Running                 0                3h26m
nvidia-container-toolkit-daemonset-8rw7p                          1/1     Running                 0                169m
nvidia-container-toolkit-daemonset-h6fm9                          1/1     Running                 0                3h35m
nvidia-container-toolkit-daemonset-mxv6h                          1/1     Running                 0                3h39m
nvidia-cuda-validator-8qzvq                                       0/1     Init:Error              6 (4m1s ago)     9m45s
nvidia-cuda-validator-mfrzg                                       0/1     Completed               0                3h33m
nvidia-cuda-validator-wzc92                                       0/1     Completed               0                3h37m
nvidia-dcgm-exporter-5bsfn                                        1/1     Running                 0                169m
nvidia-dcgm-exporter-dl2k9                                        1/1     Running                 0                3h35m
nvidia-dcgm-exporter-vzrmp                                        1/1     Running                 0                3h39m
nvidia-device-plugin-daemonset-bzhdp                              1/1     Running                 0                169m
nvidia-device-plugin-daemonset-dj7t8                              1/1     Running                 0                3h39m
nvidia-device-plugin-daemonset-ff8th                              1/1     Running                 0                3h35m
nvidia-driver-daemonset-d7cs5                                     1/1     Running                 0                3h35m
nvidia-driver-daemonset-dp6x6                                     1/1     Running                 0                3h39m
nvidia-driver-daemonset-kbds8                                     1/1     Running                 0                169m
nvidia-mig-manager-72cg2                                          1/1     Running                 0                169m
nvidia-mig-manager-8mjw6                                          1/1     Running                 0                3h39m
nvidia-mig-manager-fzd29                                          1/1     Running                 0                3h35m
nvidia-operator-validator-2qpr6                                   0/1     Init:CrashLoopBackOff   24 (4m45s ago)   169m
nvidia-operator-validator-gswwp                                   1/1     Running                 0                3h35m
nvidia-operator-validator-qsnxq                                   1/1     Running                 0                3h39m

nvidia-driver-daemonset-d7cs5.txt
nvidia-driver-daemonset-dp6x6.txt
nvidia-driver-daemonset-kbds8.txt

node with container nvidia-driver-daemonset-d7cs5 is best for me because it show me right count of GPU ( 8 x B200 )
all 3 nodes harware idetnically but not for gpu driver.

    Fabric
        State                                          : Completed
        Status                                         : Success
        CliqueId                                       : 0
        ClusterUUID                                    : 00000000-0000-0000-0000-000000000000
        Health
            Summary                                    : Healthy
            Bandwidth                                  : N/A
            Route Recovery in progress                 : N/A
            Route Unhealthy                            : N/A
            Access      Incorrect Configuration                    : N/A
            Partition Assigned                         : N/A
    Processes                                          : NoneTimeout Recovery                    : False
       
    Capabilities
        EGM                                            : disabled

additionally I collected logs again to be on the same page.
When I installed first time operator - it requires module ib_umad and containerd/config.toml to be version 4. Also was alligned.

[rahulait]

Yeah, I still see fabric in progress in logs of one of the nodes
nvidia-driver-daemonset-kbds8.txt

    Fabric
        State                                          : In Progress
        Status                                         : N/A
        CliqueId                                       : N/A
        ClusterUUID                                    : N/A
        Health
            Summary                                    : N/A
            Bandwidth                                  : N/A
            Route Recovery in progress                 : N/A
            Route Unhealthy                            : N/A
            Access Timeout Recovery                    : N/A
            Incorrect Configuration                    : N/A
            Partition Assigned                         : N/A

Also, on couple of nodes, I am seeing attached GPU count of 7 instead of 8. Is that expected?

/usr/bin/nvidia-smi --query


==============NVSMI LOG==============

Timestamp                                              : Tue Jun  2 09:49:44 2026
Driver Version                                         : 580.159.04
CUDA Version                                           : 13.0

Attached GPUs                                          : 7

Also, looking at driver install logs, driver pod on node having issue is failing to detect nvlink switches and not installing things.

On working node's driver pod logs, you'll see logs like this:

Detected NVLink5+ system
Using device mlx5_8
Started "Nvidia NVLink Subnet Manager"
Started "Nvidia Fabric Manager"

These are missing on the node having issue.

[tariq1890]

@AleksFirsta On the node where the nvidia-cuda-validator fails, can you share the outputs of the following?

• lsmod | grep mlx5
• ls /sys/class/infiniband/*/device

[AleksFirsta]

Hi @rahulait

All 3 nodes have 8 GPU + NVlink connection ( leter will install nvidia network operator) and all 3 nodes have the same list of kernel modules.
Regarding command output lsmod | grep mlx5
idential output from all 3 nodes:

lsmod|sort | grep mlx5
ib_core               507904  3 ib_umad,ib_uverbs,mlx5_ib
ib_uverbs             184320  1 mlx5_ib
macsec                 77824  1 mlx5_ib
mlx5_core            2498560  1 mlx5_ib
mlx5_ib               487424  0
mlxfw                  36864  1 mlx5_core
pci_hyperv_intf        12288  1 mlx5_core
psample                16384  1 mlx5_core
tls                   155648  2 bonding,mlx5_core
root@ai-fabric-gpu02-3dc3:~# lsmod|sort | grep mlx5|md5sum
ae2baf56323caf4b44798aaa560f99e8  -

Hi @tariq1890

from failed node

/sys/class/infiniband/mlx5_0/device:
aer_dev_correctable       infiniband_verbs  reset
aer_dev_fatal             irq               reset_method
aer_dev_nonfatal          link              resource
ari_enabled               local_cpulist     resource0
broken_parity_status      local_cpus        resource0_wc
class                     max_link_speed    revision
config                    max_link_width    rom
consistent_dma_mask_bits  mlx5_core.eth.0   sriov_drivers_autoprobe
current_link_speed        mlx5_core.rdma.0  sriov_numvfs
current_link_width        modalias          sriov_offset
d3cold_allowed            msi_bus           sriov_stride
device                    msi_irqs          sriov_totalvfs
dma_mask_bits             net               sriov_vf_device
driver                    numa_node         sriov_vf_total_msix
driver_override           pools             subsystem
enable                    power             subsystem_device
firmware_node             power_state       subsystem_vendor
hwmon                     ptp               uevent
infiniband                remove            vendor
infiniband_mad            rescan            vpd

/sys/class/infiniband/mlx5_1/device:
aer_dev_correctable       infiniband_verbs  reset
aer_dev_fatal             irq               reset_method
aer_dev_nonfatal          link              resource
ari_enabled               local_cpulist     resource0
broken_parity_status      local_cpus        resource0_wc
class                     max_link_speed    revision
config                    max_link_width    rom
consistent_dma_mask_bits  mlx5_core.eth.1   sriov_drivers_autoprobe
current_link_speed        mlx5_core.rdma.1  sriov_numvfs
current_link_width        modalias          sriov_offset
d3cold_allowed            msi_bus           sriov_stride
device                    msi_irqs          sriov_totalvfs
dma_mask_bits             net               sriov_vf_device
driver                    numa_node         sriov_vf_total_msix
driver_override           pools             subsystem
enable                    power             subsystem_device
firmware_node             power_state       subsystem_vendor
hwmon                     ptp               uevent
infiniband                remove            vendor
infiniband_mad            rescan            vpd

/sys/class/infiniband/mlx5_2/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.2  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_3/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.3  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_4/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.4  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_5/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.5  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_6/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.6  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_7/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.7  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_8/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.8  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

and good node

/sys/class/infiniband/mlx5_10/device:
aer_dev_correctable       infiniband_verbs   reset_method
aer_dev_fatal             irq                resource
aer_dev_nonfatal          link               resource0
ari_enabled               local_cpulist      resource0_wc
broken_parity_status      local_cpus         revision
class                     max_link_speed     rom
config                    max_link_width     sriov_drivers_autoprobe
consistent_dma_mask_bits  mlx5_core.rdma.10  sriov_numvfs
current_link_speed        modalias           sriov_offset
current_link_width        msi_bus            sriov_stride
d3cold_allowed            msi_irqs           sriov_totalvfs
device                    numa_node          sriov_vf_device
dma_mask_bits             pools              sriov_vf_total_msix
driver                    power              subsystem
driver_override           power_state        subsystem_device
enable                    ptp                subsystem_vendor
hwmon                     remove             uevent
infiniband                rescan             vendor
infiniband_mad            reset              vpd

/sys/class/infiniband/mlx5_11/device:
aer_dev_correctable       infiniband_verbs   reset_method
aer_dev_fatal             irq                resource
aer_dev_nonfatal          link               resource0
ari_enabled               local_cpulist      resource0_wc
broken_parity_status      local_cpus         revision
class                     max_link_speed     rom
config                    max_link_width     sriov_drivers_autoprobe
consistent_dma_mask_bits  mlx5_core.rdma.11  sriov_numvfs
current_link_speed        modalias           sriov_offset
current_link_width        msi_bus            sriov_stride
d3cold_allowed            msi_irqs           sriov_totalvfs
device                    numa_node          sriov_vf_device
dma_mask_bits             pools              sriov_vf_total_msix
driver                    power              subsystem
driver_override           power_state        subsystem_device
enable                    ptp                subsystem_vendor
hwmon                     remove             uevent
infiniband                rescan             vendor
infiniband_mad            reset              vpd

/sys/class/infiniband/mlx5_12/device:
aer_dev_correctable       infiniband_verbs   resource
aer_dev_fatal             irq                resource0
aer_dev_nonfatal          link               resource0_wc
ari_enabled               local_cpulist      revision
broken_parity_status      local_cpus         rom
class                     max_link_speed     sriov_drivers_autoprobe
config                    max_link_width     sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.12  sriov_offset
current_link_speed        modalias           sriov_stride
current_link_width        msi_bus            sriov_totalvfs
d3cold_allowed            msi_irqs           sriov_vf_device
device                    numa_node          sriov_vf_total_msix
dma_mask_bits             pools              subsystem
driver                    power              subsystem_device
driver_override           power_state        subsystem_vendor
enable                    ptp                uevent
firmware_node             remove             vendor
hwmon                     rescan             vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_13/device:
aer_dev_correctable       infiniband_verbs   resource
aer_dev_fatal             irq                resource0
aer_dev_nonfatal          link               resource0_wc
ari_enabled               local_cpulist      revision
broken_parity_status      local_cpus         rom
class                     max_link_speed     sriov_drivers_autoprobe
config                    max_link_width     sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.13  sriov_offset
current_link_speed        modalias           sriov_stride
current_link_width        msi_bus            sriov_totalvfs
d3cold_allowed            msi_irqs           sriov_vf_device
device                    numa_node          sriov_vf_total_msix
dma_mask_bits             pools              subsystem
driver                    power              subsystem_device
driver_override           power_state        subsystem_vendor
enable                    ptp                uevent
firmware_node             remove             vendor
hwmon                     rescan             vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_2/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.2  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_3/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.3  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_4/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.4  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_5/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.5  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_6/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.6  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_7/device:
aer_dev_correctable       infiniband_verbs  resource
aer_dev_fatal             irq               resource0
aer_dev_nonfatal          link              resource0_wc
ari_enabled               local_cpulist     revision
broken_parity_status      local_cpus        rom
class                     max_link_speed    sriov_drivers_autoprobe
config                    max_link_width    sriov_numvfs
consistent_dma_mask_bits  mlx5_core.rdma.7  sriov_offset
current_link_speed        modalias          sriov_stride
current_link_width        msi_bus           sriov_totalvfs
d3cold_allowed            msi_irqs          sriov_vf_device
device                    numa_node         sriov_vf_total_msix
dma_mask_bits             pools             subsystem
driver                    power             subsystem_device
driver_override           power_state       subsystem_vendor
enable                    ptp               uevent
firmware_node             remove            vendor
hwmon                     rescan            vpd
infiniband                reset
infiniband_mad            reset_method

/sys/class/infiniband/mlx5_8/device:
aer_dev_correctable       irq               resource
aer_dev_fatal             link              resource0
aer_dev_nonfatal          local_cpulist     resource0_wc
ari_enabled               local_cpus        revision
broken_parity_status      max_link_speed    rom
class                     max_link_width    sriov_drivers_autoprobe
config                    mlx5_core.rdma.8  sriov_numvfs
consistent_dma_mask_bits  modalias          sriov_offset
current_link_speed        msi_bus           sriov_stride
current_link_width        msi_irqs          sriov_totalvfs
d3cold_allowed            numa_node         sriov_vf_device
device                    pools             sriov_vf_total_msix
dma_mask_bits             power             subsystem
driver                    power_state       subsystem_device
driver_override           ptp               subsystem_vendor
enable                    remove            uevent
infiniband                rescan            vendor
infiniband_mad            reset             vpd
infiniband_verbs          reset_method

/sys/class/infiniband/mlx5_9/device:
aer_dev_correctable       irq               resource
aer_dev_fatal             link              resource0
aer_dev_nonfatal          local_cpulist     resource0_wc
ari_enabled               local_cpus        revision
broken_parity_status      max_link_speed    rom
class                     max_link_width    sriov_drivers_autoprobe
config                    mlx5_core.rdma.9  sriov_numvfs
consistent_dma_mask_bits  modalias          sriov_offset
current_link_speed        msi_bus           sriov_stride
current_link_width        msi_irqs          sriov_totalvfs
d3cold_allowed            numa_node         sriov_vf_device
device                    pools             sriov_vf_total_msix
dma_mask_bits             power             subsystem
driver                    power_state       subsystem_device
driver_override           ptp               subsystem_vendor
enable                    remove            uevent
infiniband                rescan            vendor
infiniband_mad            reset             vpd
infiniband_verbs          reset_method

/sys/class/infiniband/mlx5_bond_0/device:
aer_dev_correctable       infiniband_verbs  reset
aer_dev_fatal             irq               reset_method
aer_dev_nonfatal          link              resource
ari_enabled               local_cpulist     resource0
broken_parity_status      local_cpus        resource0_wc
class                     max_link_speed    revision
config                    max_link_width    rom
consistent_dma_mask_bits  mlx5_core.eth.0   sriov_drivers_autoprobe
current_link_speed        mlx5_core.rdma.0  sriov_numvfs
current_link_width        modalias          sriov_offset
d3cold_allowed            msi_bus           sriov_stride
device                    msi_irqs          sriov_totalvfs
dma_mask_bits             net               sriov_vf_device
driver                    numa_node         sriov_vf_total_msix
driver_override           pools             subsystem
enable                    power             subsystem_device
firmware_node             power_state       subsystem_vendor
hwmon                     ptp               uevent
infiniband                remove            vendor
infiniband_mad            rescan            vpd

[rahulait]

All 3 nodes have 8 GPU + NVlink connection

Can you check the output of nvidia-smi and confirm with it? I saw 7 in couple of nodes in the previous outputs shared.

Can you run this on each node and send its output as well.

for d in /sys/class/infiniband/mlx5_*; do
  echo "===== $d ====="
  echo -n "BDF: "; readlink -f "$d/device" | awk -F/ '{print $NF}'
  echo -n "vendor: "; cat "$d/device/vendor" 2>/dev/null
  echo -n "device: "; cat "$d/device/device" 2>/dev/null
  echo -n "subsystem_vendor: "; cat "$d/device/subsystem_vendor" 2>/dev/null
  echo -n "subsystem_device: "; cat "$d/device/subsystem_device" 2>/dev/null
  echo -n "current_link_speed: "; cat "$d/device/current_link_speed" 2>/dev/null
  echo -n "current_link_width: "; cat "$d/device/current_link_width" 2>/dev/null
  for p in "$d"/ports/*; do
    [ -d "$p" ] || continue
    echo "--- port $(basename "$p") ---"
    echo -n "state: "; cat "$p/state" 2>/dev/null
    echo -n "phys_state: "; cat "$p/phys_state" 2>/dev/null
    echo -n "link_layer: "; cat "$p/link_layer" 2>/dev/null
    echo -n "rate: "; cat "$p/rate" 2>/dev/null
  done
done

Based on your previous shared logs:

broken ai-fabric-gpu02-3dc3:
  attached_gpus=7
  missing GPU BDF: E4:00.0
  Fabric State: In Progress
  CliqueId: N/A
  NVLink active links: 126
  NVLink Tx/Rx packets: 0 / 0
  hard NVLink errors: 0
  Fabric Manager: NV_ERR_INVALID_DEVICE
  missing mlx5 BDFs vs working reports:
    e3:00.0, e3:00.1, e3:00.2, e3:00.3, e5:00.0

working ai-fabric-gpu01-3dc4:
  attached_gpus=7
  missing GPU BDF: 94:00.0
  Fabric State: Completed
  CliqueId: 0
  NVLink Tx/Rx packets: nonzero
  hard NVLink errors: 0

working ai-fabric-gpu01-3dc3:
  attached_gpus=8
  missing GPU BDF: none
  Fabric State: Completed
  CliqueId: 0
  NVLink Tx/Rx packets: nonzero
  hard NVLink errors: 0

The broken node differs at device/topology enumeration level, not kernel-module level.

The most suspicious delta is now both:

GPU E4:00.0 missing on broken node
mlx5 e3/e5-side devices missing from broken node bug-report dmesg

That makes this look like a platform/topology enumeration issue, possibly hardware/firmware/PCIe/NVL5 management path related, rather than “GPU Operator did not load mlx5 modules.”

[AleksFirsta]

@rahulait ,

We have dump of OS Ubuntu 24.04 (hpcm) with preinstalled drivers manuallly ( Os run in RAM ) all components and output of Nvidia-smi show me 8 GPU ( 0 - 7 ) on each node. Today I did run this OS in RAM to exclude hardware issue.
Its not hardware for sure. just to share with you it.

node1.txt
node2.txt
node3(problem).txt

[rahulait]

The RAM-boot Ubuntu test is good evidence that the hardware can enumerate 8 GPUs. However, in the GPU Operator environment, the NVIDIA stack only enumerates 7 GPUs and fabric initialization never completes. That points to a software/boot-stack difference: installed OS kernel/PCIe enumeration, GPU Operator driver container, driver/Fabric Manager versioning, or host preinstalled NVIDIA component conflict.

The biggest thing I’d now check is whether the installed OS has any host NVIDIA/Fabric Manager packages fighting with GPU Operator. Earlier bug report had a suspicious Fabric Manager mismatch:

Fabric Manager interface version 595.71.05
driver version 580.126.20

That may be stale log history, but wanted to check if you had manually preinstalled drivers on the installed OS or there were retries done with other driver versions.

To compare the failing environment directly, please switch back to the installed OS where GPU Operator is running and collect the same NVIDIA bug report again from that environment. We want to confirm what the installed OS + GPU Operator driver stack is detecting now: GPU count, missing BDF if any, Fabric state, NVLink traffic, and Fabric Manager/NVLSM errors.

Please collect from the affected node after booting back into the installed OS and after the GPU Operator driver pod has started.

[AleksFirsta]

dpkg -l|grep nvidia

Doesnt show me any packages on workers.

We have node 4 ( No k8s cluster with RAM OS )

dpkg -l|grep nvidia
ii  libnvidia-cfg1:amd64                             590.48.01-0ubuntu1                        amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common                                 590.48.01-0ubuntu1                        all          Shared files used by the NVIDIA libraries
ii  libnvidia-compute:amd64                          590.48.01-0ubuntu1                        amd64        NVIDIA libcompute package
ii  libnvidia-container-tools                        1.18.2-1                                  amd64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64                       1.18.2-1                                  amd64        NVIDIA container runtime library
ii  libnvidia-decode:amd64                           590.48.01-0ubuntu1                        amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-egl-gbm1:amd64                         1.1.3-1ubuntu1                            amd64        GBM EGL external platform library for NVIDIA
ii  libnvidia-egl-wayland1:amd64                     1:1.1.21-1ubuntu1                         amd64        Wayland EGL External Platform library -- shared library
ii  libnvidia-egl-xcb1:amd64                         1.0.4-1ubuntu1                            amd64        This is an EGL platform library for the NVIDIA driver to support
ii  libnvidia-egl-xlib1:amd64                        1.0.4-1ubuntu1                            amd64        This is an EGL platform library for the NVIDIA driver to support
ii  libnvidia-encode:amd64                           590.48.01-0ubuntu1                        amd64        NVENC Video Encoding runtime library
ii  libnvidia-extra:amd64                            590.48.01-0ubuntu1                        amd64        Extra libraries for the NVIDIA driver
ii  libnvidia-fbc1:amd64                             590.48.01-0ubuntu1                        amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl:amd64                               590.48.01-0ubuntu1                        amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-gpucomp:amd64                          590.48.01-0ubuntu1                        amd64        NVIDIA binary GPU compiler library
ii  libnvidia-nscq                                   590.48.01-0ubuntu1                        amd64        NVSwitch Configuration and Query library
ii  nvidia-container-toolkit                         1.18.2-1                                  amd64        NVIDIA Container toolkit
ii  nvidia-container-toolkit-base                    1.18.2-1                                  amd64        NVIDIA Container Toolkit Base
ii  nvidia-dkms-open                                 590.48.01-0ubuntu1                        amd64        NVIDIA DKMS package (open kernel module)
ii  nvidia-driver-open                               590.48.01-0ubuntu1                        amd64        NVIDIA driver (open kernel) metapackage
ii  nvidia-fabricmanager                             590.48.01-0ubuntu1                        amd64        Fabric Manager for NVSwitch based systems
ii  nvidia-firmware                                  590.48.01-0ubuntu1                        amd64        Firmware files used by the kernel module
ii  nvidia-fs                                        2.27.3-1                                  amd64        NVIDIA filesystem for GPUDirect Storage
ii  nvidia-fs-dkms                                   2.27.3-1                                  amd64        NVIDIA GPUDirect Storage driver
ii  nvidia-gds                                       13.1.1-1                                  amd64        GPU Direct Storage meta-package
ii  nvidia-gds-13-1                                  13.1.1-1                                  amd64        GPU Direct Storage 13.1 meta-package
ii  nvidia-imex                                      590.48.01-0ubuntu1                        amd64        IMEX Manager for NVIDIA based systems
ii  nvidia-kernel-common                             590.48.01-0ubuntu1                        amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-open                        590.48.01-0ubuntu1                        amd64        NVIDIA kernel source package
ii  nvidia-modprobe                                  590.48.01-0ubuntu1                        amd64        Load the NVIDIA kernel driver and create device files
ii  nvidia-open                                      590.48.01-0ubuntu1                        amd64        NVIDIA Driver meta-package, Open GPU kernel modules, latest version
ii  nvidia-persistenced                              590.48.01-0ubuntu1                        amd64        daemon to maintain persistent software state in the NVIDIA driver
ii  nvidia-settings                                  590.48.01-0ubuntu1                        amd64        Tool for configuring the NVIDIA graphics driver
ii  screen-resolution-extra                          0.18.3ubuntu0.24.04.1                     all          Extension for the nvidia-settings control panel
ii  xserver-xorg-video-nvidia                        590.48.01-0ubuntu1                        amd64        NVIDIA binary Xorg driver

must_garther. cant be run - there to kubectl tool and not in k8s.

node4(no k8s).txt

[rahulait]

Agreed, let’s focus only on the affected Kubernetes worker.

Since the issue is on ai-fabric-gpu02-3dc3, please boot that same node back into the installed OS (not into RAM OS) where GPU Operator is running and collect a fresh must-gather from that cluster again.

The RAM-boot output from another node (node4) does not help us confirm what the failing node’s GPU Operator stack is detecting. We need the report from the affected node and affected environment only.

Specifically we want to confirm on ai-fabric-gpu02-3dc3:

• whether NVIDIA still sees 7 or 8 GPUs
• whether BDF E4:00.0 is still missing
• whether Fabric state is still In Progress
• whether NVLink Tx/Rx is still zero
• whether Fabric Manager/NVLSM still reports NV_ERR_INVALID_DEVICE

[AleksFirsta]

Please check

k get nodes -o wide
NAME                          STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION              CONTAINER-RUNTIME
ai-fabric-gpu01-3dc3          Ready    worker          23h   v1.36.1   10.113.1.50   <none>        Ubuntu 24.04.4 LTS   6.8.0-124-generic (amd64)   containerd://2.3.0
ai-fabric-gpu01-3dc4          Ready    worker          23h   v1.36.1   10.123.1.60   <none>        Ubuntu 24.04.4 LTS   6.8.0-124-generic (amd64)   containerd://2.3.0
ai-fabric-gpu02-3dc3          Ready    worker          23h   v1.36.1   10.113.1.51   <none>        Ubuntu 24.04.4 LTS   6.8.0-124-generic (amd64)   containerd://2.3.0
ai-fabric-k8s-master01-3dc3   Ready    control-plane   24h   v1.36.1   10.113.1.12   <none>        Ubuntu 24.04.4 LTS   6.8.0-124-generic (amd64)   containerd://2.3.0
ai-fabric-k8s-master02-3dc4   Ready    control-plane   24h   v1.36.1   10.123.1.22   <none>        Ubuntu 24.04.4 LTS   6.8.0-124-generic (amd64)   containerd://2.3.0
ai-fabric-k8s-master03-3dc2   Ready    control-plane   23h   v1.36.1   10.125.1.20   <none>        Ubuntu 24.04.4 LTS   6.8.0-124-generic (amd64)   containerd://2.3.0

[AleksFirsta]

gpu_device_usage_ai-fabric-gpu01-3dc4.log
must-gather.log
must-gather.stderr.log
gpu_device_usage_ai-fabric-gpu02-3dc3.log
gpu_device_usage_ai-fabric-gpu01-3dc3.log
nvidia-bug-report_ai-fabric-gpu01-3dc4.log.gz
nvidia-bug-report_ai-fabric-gpu02-3dc3.log.gz
nvidia-bug-report_ai-fabric-gpu01-3dc3.log.gz
gpu_operand_ds.yaml
gpu_operand_pod_nvidia-operator-validator-nw56h.previous.log
gpu_operand_pod_nvidia-mig-manager-t7jk9.log
gpu_operand_pod_nvidia-mig-manager-t7jk9.previous.log
gpu_operand_pod_nvidia-mig-manager-z6rqt.log
gpu_operand_pod_nvidia-mig-manager-z6rqt.previous.log
gpu_operand_pod_nvidia-operator-validator-hpnzk.log
gpu_operand_pod_nvidia-operator-validator-hpnzk.previous.log
gpu_operand_pod_nvidia-operator-validator-lm8z6.log
gpu_operand_pod_nvidia-operator-validator-lm8z6.previous.log
gpu_operand_pod_nvidia-operator-validator-nw56h.log
gpu_operand_pod_nvidia-device-plugin-daemonset-mq2f8.log
gpu_operand_pod_nvidia-device-plugin-daemonset-mq2f8.previous.log
gpu_operand_pod_nvidia-driver-daemonset-4sxph.log
gpu_operand_pod_nvidia-driver-daemonset-4sxph.previous.log
gpu_operand_pod_nvidia-driver-daemonset-8dr2w.log
gpu_operand_pod_nvidia-driver-daemonset-8dr2w.previous.log
gpu_operand_pod_nvidia-driver-daemonset-kw7kb.log
gpu_operand_pod_nvidia-driver-daemonset-kw7kb.previous.log
gpu_operand_pod_nvidia-mig-manager-cwcnq.log
gpu_operand_pod_nvidia-mig-manager-cwcnq.previous.log
gpu_operand_pod_nvidia-cuda-validator-msrcm.previous.log
gpu_operand_pod_nvidia-dcgm-exporter-8wsfn.log
gpu_operand_pod_nvidia-dcgm-exporter-8wsfn.previous.log
gpu_operand_pod_nvidia-dcgm-exporter-f7hqd.log
gpu_operand_pod_nvidia-dcgm-exporter-f7hqd.previous.log
gpu_operand_pod_nvidia-dcgm-exporter-lp6qz.log
gpu_operand_pod_nvidia-dcgm-exporter-lp6qz.previous.log
gpu_operand_pod_nvidia-device-plugin-daemonset-9m6sz.log
gpu_operand_pod_nvidia-device-plugin-daemonset-9m6sz.previous.log
gpu_operand_pod_nvidia-device-plugin-daemonset-bpf4s.log
gpu_operand_pod_nvidia-device-plugin-daemonset-bpf4s.previous.log
gpu_operand_pod_nvidia-container-toolkit-daemonset-2wpzc.log
gpu_operand_pod_nvidia-container-toolkit-daemonset-2wpzc.previous.log
gpu_operand_pod_nvidia-container-toolkit-daemonset-bdrtb.log
gpu_operand_pod_nvidia-container-toolkit-daemonset-bdrtb.previous.log
gpu_operand_pod_nvidia-container-toolkit-daemonset-s7tz2.log
gpu_operand_pod_nvidia-container-toolkit-daemonset-s7tz2.previous.log
gpu_operand_pod_nvidia-cuda-validator-k8ns6.log
gpu_operand_pod_nvidia-cuda-validator-k8ns6.previous.log
gpu_operand_pod_nvidia-cuda-validator-mnkgn.log
gpu_operand_pod_nvidia-cuda-validator-mnkgn.previous.log
gpu_operand_pod_nvidia-cuda-validator-msrcm.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-78kjd.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-78kjd.previous.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-fj422.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-fj422.previous.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-gtxgs.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-gtxgs.previous.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-m92bw.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-m92bw.previous.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-nj9qh.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-nj9qh.previous.log
gpu_operand_pod_gpu-feature-discovery-gknb4.log
gpu_operand_pod_gpu-feature-discovery-gknb4.previous.log
gpu_operand_pod_gpu-feature-discovery-pxqpc.log
gpu_operand_pod_gpu-feature-discovery-pxqpc.previous.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-gc-68f5b5c7nsq72.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-gc-68f5b5c7nsq72.previous.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-master-86f5x2g5b.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-master-86f5x2g5b.previous.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-4dqlv.log
gpu_operand_pod_gpu-operator-1780411155-node-feature-discovery-worker-4dqlv.previous.log
gpu_operand_pod_gpu-feature-discovery-2dww5.log
gpu_operand_pod_gpu-feature-discovery-2dww5.previous.log
gpu_operand_pod_images.txt
gpu_operand_pods.yaml
gpu_operator_pod.log
gpu_operator_pod.previous.log
gpu_operator_pod.yaml
cluster_policy.yaml

[rahulait]

Can you also attach the gpu_operand_pods.status file just to see current status of pods in cluster? Attaching entire dir after zipping it helps in getting all logs in one file.