Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions docs/user/cluster-management/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -593,6 +593,106 @@ The NVIDIA Cluster Agent supports various feature flags that can be enabled or d
| SelfHosted | Enables local vault-based authentication for self-hosted deployments. Required when `ngcConfig.clusterSource` is `self-managed`. |
| HelmAllowCPUNodes | Allow CPU-only pods (e.g. etcd, redis, envoy) from Helm-based functions to be scheduled on non-GPU nodes; GPU pods keep required instance-type affinity. Reduces cost and improves GPU utilization. Mutually exclusive with HelmResourceConstraints. See [helm-allow-cpu-nodes](./configuration.md). |

### Helm Workload Resource Limits

Self-managed NVCF enables CPU and memory resource-limit validation for Helm
workloads in self-managed stack `v0.7.0` and later. Standalone compute-plane
stack `v0.2.0` is the first compute-plane release with the same default policy.

NVCA introduced `EnforceHelmFunctionResourceLimits` in `v2.47.0` and
`EnforceHelmTaskResourceLimits` in `v2.47.1`. These NVCA feature gates default
to disabled in the NVCA binary. The self-managed and compute-plane stacks
enable them by default. Upgrading the NVCA binary to `v3.8.0` does not activate
this policy by itself.

Before upgrading across either stack boundary, add CPU and memory limits to
every regular container and init container rendered by each Helm function or
Helm task chart. For example:

```yaml
resources:
limits:
cpu: "1"
memory: 1Gi
```

If a rendered container omits either limit, the NVCA validating webhook rejects
the workload. The error includes text similar to:

```text
admission webhook "validate-helm-charts.nvca.nvcf.nvidia.io" denied the request: container <name> has no resource limits
```

The following feature gates control the checks independently:

- `EnforceHelmFunctionResourceLimits` validates Helm function charts.
- `EnforceHelmTaskResourceLimits` validates Helm task charts.

#### Temporarily Disable Enforcement

Adding limits is the durable fix. If legacy charts cannot be updated before an
upgrade, add a negative override to the target compute-plane environment file:

```yaml
global:
nvcaOperator:
selfManaged:
featureGateValues:
- "-EnforceHelmFunctionResourceLimits"
- "-EnforceHelmTaskResourceLimits"
```

The leading `-` is required. Include only the negative override for each check
that you need to disable. The stack preserves negative overrides while adding
its other default feature gates.

Disabling `HelmResourceConstraints` does not disable these CPU and memory
checks. Use the two enforcement gates shown above.

<Warning>
Use negative overrides only as a temporary compatibility measure. Workloads
without CPU and memory limits can cause resource contention and pod eviction.
Add limits to the charts, remove the negative overrides, and apply the
compute-plane configuration again.
</Warning>

#### Apply and Verify the Override

From the repository root, render and apply the target compute-plane stack. Use
the same cluster name, environment, kubeconfig, and Kubernetes context as the
existing installation:

```bash
make -C deploy/stacks/nvcf-compute-plane template \
CLUSTER_NAME=<cluster-name> \
HELMFILE_ENV=<environment-name> \
KUBECONFIG_FILE=<absolute-path-to-kubeconfig> \
COMPUTE_KUBE_CONTEXT=<kubernetes-context>

make -C deploy/stacks/nvcf-compute-plane apply \
CLUSTER_NAME=<cluster-name> \
HELMFILE_ENV=<environment-name> \
KUBECONFIG_FILE=<absolute-path-to-kubeconfig> \
COMPUTE_KUBE_CONTEXT=<kubernetes-context>
```

Wait for the NVCA Operator rollout and inspect the desired and applied feature
gates:

```bash
kubectl --kubeconfig <absolute-path-to-kubeconfig> \
--context <kubernetes-context> \
-n nvca-operator rollout status deployment/nvca-operator --timeout=10m

kubectl --kubeconfig <absolute-path-to-kubeconfig> \
--context <kubernetes-context> \
-n nvca-operator get nvcfbackend <cluster-name> \
-o jsonpath='{.spec.featureGate.values}{"\n"}{.status.featureGate.values}{"\n"}'
```

Both lines must contain the expected positive gates when enforcement is active,
or the expected negative gates when the compatibility override is active.

### Setting Feature Flags at Install Time

Feature flags can be set during the initial NVCA Operator installation through Helm values.
Expand Down
Loading