[Bug]: nvidia.com/gpu.product label removed when MIG repartitioning remains in pending state

[anoopsinghnegi]

Environment

• GPU Operator Version: 26.3.0
• GPU Model: H100-NVL
• MIG Strategy: single

Description

During MIG repartitioning operations in NVIDIA GPU Operator, if the MIG Manager is unable to complete the requested MIG partitioning and the node remains stuck in:

mig.config.state=pending

the nvidia.com/gpu.product label gets removed from the node.

One reproducible scenario observed is:

• MIG repartitioning starts
• node reboots during repartitioning
• MIG Manager cannot complete reconciliation after reboot
• mig.config.state remains pending
• nvidia.com/gpu.product label gets removed

This causes issues for external components/controllers that rely on the nvidia.com/gpu.product label to identify the GPU type for reconciliation, scheduling, or inventory purposes.

Current Behavior

When repartitioning is interrupted or fails and the node remains in pending state:

• mig.config.state=pending
• nvidia.com/gpu.product label is removed entirely

As a result, dependent components lose visibility into the underlying GPU hardware type.

Expected Behavior

Instead of removing the label completely during repartitioning/pending state, preserve the GPU product information and append a suffix indicating the node is still pending repartitioning.
similar to an invalid partitioning case, the label is set to nvidia.com/gpu.product=NVIDIA-H200-NVL-INVALID

Example:

nvidia.com/gpu.product=NVIDIA-H200-NVL-ERROR

This would:

• preserve GPU hardware visibility
• indicate repartitioning is incomplete/pending
• help external controllers continue identifying the GPU type
• avoid reconciliation or scheduling issues caused by label disappearance

Impact

External components depending on nvidia.com/gpu.product cannot reliably determine GPU type once the label is removed, especially during interrupted repartitioning scenarios such as node reboot.

[rajathagasthya]

@anoopsinghnegi Thanks for the detailed report! This looks to be an edge case when nodes reboot during repartitioning. Could you clarify a few things so any possible fix lands in the right component?

1. When mig.config.state=pending and the label disappears, what's the state of the driver pod? Does nvidia-smi return any devices?
2. Is the gpu-feature-discovery pod on that node running, paused (nvidia.com/gpu.deploy.gpu-feature-discovery=paused-for-mig-change), or crash-looping? GFD logs around the time the label was removed would help.
3. Could you share the full set of node labels and the NodeFeature CR for the affected node when it's in the stuck state?

Context for why I'm asking: the nvidia.com/gpu.product label is owned by GFD (in k8s-device-plugin), and the -INVALID suffix you referenced is add only when MIG mode is enabled but slices aren't configured, not when devices can't be enumerated. If GFD is seeing zero devices, it returns no labels and NFD garbage-collects the existing ones, which matches what you're seeing. This is a bit tricky since GFD would have to persist last-known state across restart, which it's not designed to do currently.

[myeolenv]

Will mark it as bug once we get addition requested information and able to reproduce this as bug.