At the heart of the AI revolution are GPUs and the platform that provides access to them is Kubernetes. Workloads historically access GPUs and other devices with the device plugin API but features are lacking. The new Dynamic Resource Allocation (DRA) feature helps maximize GPU utilization across workloads with additional features like the ability to control device sharing across Pods, use multiple GPU models per node, handle dynamic allocation of multi-instance GPU (MIG) and more. DRA is not limited to GPUs but any specialized hardware that a Pod may use including network attached resources such as edge devices like IP cameras. DRA is a new way to request for resources like GPUs and gives the ability to precisely control how resources are shared between Pods. This tutorial introduces DRA, reviews the “behind-the-scenes” of DRA in the Kubernetes cluster and walks through multiple ways to use DRA to request for GPU and a network attached resource.
In this tutorial we will install a Kubernetes cluster, review the DRA resources and how they work, install a sample DRA driver, run workloads that use the DRA driver.
Module 1 - Introduction to Dynamic Resource Allocation 10 minutes
Module 2 - DRA Under the Covers 15 minutes
Module 3 - A Look into DRA Drivers 15 minutes
Module 4 - Deploy a DRA Driver and Workloads (25 minutes)
- Deploy a DeviceClass
- Create RBAC Authorization for the DRA Driver
- PriorityClass
- Deploy the DRA Driver
- ResourceSlice
- Ollama Pod Workloads
- Sharing a ResoureClaim
- Job with ResourceClaimTemplate
- Additional DRA Drivers
- DRA Driver for CPU Resources
- AMD
- FuriosaAI
Kubernetes v1.34 was released in August and the core components of Dynamic Resource Allocation were promoted to stable / GA. Workloads need more than CPU and memory but also need specialized hardware. DRA is a new API for Pods to request and access specialized hardware like accelerators such as GPUs, fabric-attached GPUs, Tensor Processing Unit (TPU) or network-attached devices. Support for hardware are provided by vendors via DRA drivers.
The previous way of accessing specialized hardware was with device plugins which had limitations such as the inability to share allocated devices among multiple Pods and the device had to be attached to a node (node-local) not across the network fabric. Device plugins are good for requesting single, linear quantity of resources. The evolving nature of resources and extension of Kubernetes workloads to use specialized resources required a more flexible way to access and allocate those resources.
Compared to device plugins, DRA offers the following benefits to use specialized devices by Kubernetes workloads:
- device filtering with Common Expression Language (CEL) for fine-grained filtering
- device sharing with multiple Pods
- centralized device categorization
- flexible configuration like dynamic GPU partitioning
- accelerators and other devices as well like NICs, FPGAs, network-attached devices
- pass configuration parameters to devices
- Pod is scheduled when required resources are available
- more features coming
The DRA driver (installed locally) has a component called the kubelet-plugin (typically a DaemonSet) that talks to the node's kubelet about the device and prepares the device to be used.
The DRA driver publishes the available device in the form of a ResourceSlice object which is tied to the specific node where the DRA driver is installed -- so when a node goes down, the corresponding ResourceSlices will also be gone.
A DeviceClass is similar to a StorageClass. A DeviceClass has device configurations and parameters that are used as selectors. A DeviceClass corresponds to a device driver and is installed by the cluster admin or created by the DRA driver.
When a Pod wants to use a device, the user creates a ResourceClaim or uses a ResourceClaimTemplate that references a DeviceClass.
A cluster admin installs a corresponding DeviceClass that has device configuration and selectors for the device.
Since DRA was GA'd in Kubernetes v1.34 released in August of 2025. You will use kind (Kubernetes in Docker) in this lab.
The VMs provided are 2 CPU, 8 GB RAM, RHEL 9
Go to the workshop platform to obtain login instructions to a VM:
SSH into your VM: Note: the following is an example, use the hostname provided to you in the above
ssh lab-user@rhel9.m4kkb.sandbox1943.opentlc.comAccept to continue to connect if prompted and enter the provided password when prompted
The authenticity of host 'rhel9.m4kkb.sandbox1943.opentlc.com (3.22.88.9)' can't be established.
ED25519 key fingerprint is SHA256:1qrBdHzv0B3ExYHpdKBMcNOQwbpyzKyTB5Y4S3kjlfQ.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'rhel9.m4kkb.sandbox1943.opentlc.com' (ED25519) to the list of known hosts.
lab-user@rhel9.m4kkb.sandbox1943.opentlc.com's password: There is a script that will install kind cluster with 1 control plane and 1 worker node cluster.
Run the install script: Note: the install script will take about 3.5 minutes and works on RHEL 9
curl https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/install-kind.sh | shOutput:
This script installs Docker, kind, kubectl, and creates a kind cluster
This will take several minutes to complete
Installing docker
Updating Subscription Management repositories.
Unable to read consumer identity
...
kubectl: OK
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.34.0) 🖼
✓ Preparing nodes 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11928 100 11928 0 0 99400 0 --:--:-- --:--:-- --:--:-- 99400
Downloading https://get.helm.sh/helm-v3.19.0-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
kind cluster is readyTest the cluster:
kubectl versionOutput:
Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.34.0Check the cluster's nodes:
kubectl get nodesOutput:
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane 2m27s v1.34.0
kind-worker Ready <none> 2m14s v1.34.0DRA graduated to stable in v1.34 on August 27, 2025.
In v1.34, the core DRA APIs resource.k8s.io/v1 graduated to stable from resource.k8s.io/v1beta1.
A DRA driver has 2 components that coordinate with each other
- node-local kubelet plugin (DaemonSet) on nodes with the advertised device(s)
- centralized controller running in HA (deployment)
Centralized Controller
- performs the ResourceClaim creation from ResourceClaimTemplates
- performs the actual ResourceClaim allocation after a node is selected by the scheduler
- performs deallocation of ResourceClaim once deleted
Node-local kubelet plugin
- advertises the node-local state that the centralize controller needs to help make allocation decisions
- makes node-local operations required to prepare a ResourceClaim (parameters may need to be setup) or deallocate a ResourceClaim on a node
- pass the device associated with prepared ResourceClaim to the kubelet which will then forward to the container runtime
Note: There is also a scheduler plugin that detects Pods which references a ResourceClaim or ResourceClaimTemplate and ensures the resource is allocated where the Pod is scheduled to
The DRA Driver has two main components:
- centralized controller: manages resource allocation requests
- node-local kubelet-plugin: handles resource allocation on the node, typically installed as a DaemonSet and may use node affinity to schedule DaemonSets appropriately e.g.
feature.node.kubernetes.io/pci-10de.present=true,feature.node.kubernetes.io/cpu-model.vendor_id=NVIDIAornvidia.com/gpu.present=true
ResourceSlice: created by the DRA driver, tied to the node, represents devices represented by the driver on the node. Used by Kubernetes to find nodes with devices for scheduling Pods to nodes that can access the requested resource(s)
DeviceClass: defines category of devices e.g. gpu.nvidia.com. Parameters in DeviceClasses match to a device(s) in ResourceSlices
ResourceClaim: request for specific devices from a DeviceClass
ResourceClaims can be referenced by multiple Pods if the device can be shared and not tied to any Pod's lifecycle
ResourceClaimTemplate: template to generate per-Pod ResourceClaims. When a ResourceClaim is created from a ResourceClaimTemplate, it is bound to the Pod's lifecycle
Represents the available devices represented by a driver on the node.
The DRA driver creates the ResourceSlice.
The Kubernetes scheduler uses ResourceSlices to determine where to allocate Pods.
kubectl explain resourceslice | head -n 30Output:
GROUP: resource.k8s.io
KIND: ResourceSlice
VERSION: v1
DESCRIPTION:
ResourceSlice represents one or more resources in a pool of similar
resources, managed by a common driver. A pool may span more than one
ResourceSlice, and exactly how many ResourceSlices comprise a pool is
determined by the driver.
At the moment, the only supported resources are devices with attributes and
capacities. Each device in a given pool, regardless of how many
ResourceSlices, must have a unique name. The ResourceSlice in which a device
gets published may change over time. The unique identifier for a device is
the tuple <driver name>, <pool name>, <device name>.
Whenever a driver needs to update a pool, it increments the
pool.Spec.Pool.Generation number and updates all ResourceSlices with that
new number and new resource definitions. A consumer must only use
ResourceSlices with the highest generation number and ignore all others.
When allocating all resources in a pool matching certain criteria or when
looking for the best solution among several different alternatives, a
consumer should check the number of ResourceSlices in a pool (included in
each ResourceSlice) to determine whether its view of a pool is complete and
if not, should wait until the driver has completed updating the pool.
For resources that are not local to a node, the node name is not set.
Instead, the driver may use a node selector to specify where the devices are
available.Within a ResourceSlice are the following:
- Resource pool: a group of 1 or more resources the driver manages
- Devices: are devices in a managed pool, devices lists information like versions, capacity, or other attributes
- Nodes: the nodes that can access the resources
Here's an example of a ResourceSlice:
apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
name: nvidia-gpu-slice
spec:
driver: "gpu.nvidia.com" # The name of the DRA driver managing these resources
pool:
name: "nvidia-gpu-pool" # A logical pool of resources managed by the driver
generation: 1 # Incremented by the driver when the pool configuration changes
resourceSliceCount: 1 # Total number of ResourceSlices in this pool
nodeName: "node-1" # The name of the node where these resources are located
devices:
- name: "gpu-0" # Unique name for the device within the pool
attributes:
vendor:
string: "NVIDIA"
model:
string: "A100"
capacity:
memory:
value: "40Gi" # Example capacity: 40 GiB of memoryThe DeviceClass resource correspondes a resource driver with a named resource in the cluster.
Contains pre-defined selection criteria for certain devices and configuration for them.
Can include optional parameters like a GPUClaimParameters that we'll look at later.
Each request to allocate a device in a ResourceClaim must reference exactly one DeviceClass.
A DeviceClas defines a category of devices.
The DeviceClass may be installed with the driver.
Use kubectl explain see the DeviceClass description:
kubectl explain deviceclass | head -n 8Output:
GROUP: resource.k8s.io
KIND: DeviceClass
VERSION: v1
DESCRIPTION:
DeviceClass is a vendor- or admin-provided resource that contains device
configuration and selectors. It can be referenced in the device requests of
a claim to apply these presets. Cluster scoped.An example DeviceClass manifest is:
apiversion: resource.k8s.io/v1alpha3
kind: DeviceClass
metadata:
name: gpu.vendor.com
spec:
selectors:
- cel:
expression: "device.driver == 'gpu.vendor.com'"Under the DeviceClass .spec.selectors, a Common Expression Language (CEL) expression is used to select a device.
You can use multiple CEL expressions e.g. device.driver == 'gpu.vendor.com' && device.attributes['gpu.vendor.com'].type == 'gpu' && device.attributes['gpu.vendor.com'].accelerator-type == 'high-performance'
We'll look into Intel and NVIDIA's DRA resources later but for now, Intel's DeviceClass is the following:
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: gpu.intel.com
spec:
selectors:
- cel:
expression: device.driver == "gpu.intel.com"
extendedResourceName: intel.com/gpuAn extendedResourceName is an additional way for Pods to request resources.
It is in alpha as of v1.34.
From kubectl explain deviceclass.spec.extendedResourceName:
ExtendedResourceName is the extended resource name for the devices of this
class. The devices of this class can be used to satisfy a pod's extended
resource requests. It has the same format as the name of a pod's extended
resource. It should be unique among all the device classes in a cluster. If
two device classes have the same name, then the class created later is
picked to satisfy a pod's extended resource requests. If two classes are
created at the same time, then the name of the class lexicographically
sorted first is picked.A ResourceClaim describes a request for access to resources in the cluster, for use by workloads.
For example, if a workload needs an accelerator device with specific properties, this is how that request is expressed.
The status stanza tracks whether this claim has been satisfied and what specific resources have been allocated.
A ResourceClaim is a claim to use a specific DeviceClass and represents an actual resource allocation made by the resource driver.
Users create ResourceClaims and refer to the DeviceClass they want to allocate resources for. The Pod can then use these resources with a ResourceClaim.
Let's use kubectl explain to see the ResourceClaim's description:
kubectl explain resourceclaim | head -n 10Output:
GROUP: resource.k8s.io
KIND: ResourceClaim
VERSION: v1
DESCRIPTION:
ResourceClaim describes a request for access to resources in the cluster,
for use by workloads. For example, if a workload needs an accelerator device
with specific properties, this is how that request is expressed. The status
stanza tracks whether this claim has been satisfied and what specific
resources have been allocated.ResourceClaims can be created manually by users or by Kubernetes from a ResourceClaimTemplate.
If a ResourceClaimTemplate is used then the ResourceClaim is bound to the Pod that uses the ResourceClaimTemplate and bound to the Pod's lifecycle.
User created (not-from-ResourceClaimTemplate) ResourceClaims can be shared among multiple Pods and are not bound to any Pod's lifecycle.
Example ResourceClaim:
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
name: shared-gpu-resourceclaim
spec:
devices:
requests:
- name: gpu-claim
exactly:
deviceClassName: gpu.vendor.com
allocationMode: All
selectors:
- cel:
expression: |-
device.attributes["driver.vendor.com"].type == "gpu" &&
device.capacity["driver.vendor.com"].memory == quantity("80Gi")This ResourceClaim Ceates requests devices in the gpu.vendor.com DeviceClass that matches both of the following parameters:
- Devices that have a
driver.vendor.com/typeattribute with a value of gpu. - Devices that have 80Gi of capacity.
A ResourceClaimTemplate is a template to create ResourceClaims from, for per-Pod access to resources.
Let's use kubectl explain to see the ResourceClaimTemplate's description:
kubectl explain resourceclaimtemplate | head -n 6Output:
GROUP: resource.k8s.io
KIND: ResourceClaimTemplate
VERSION: v1
DESCRIPTION:
ResourceClaimTemplate is used to produce ResourceClaim objects.ResourceClaimTemplates may look similar to ResourceClaims since they are used to generate ResourceClaims. Here's an example ResourceClaimTemplate that would produce the example ResourceClaim above:
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: resourceclaimtemplate-for-per-pod-resourceclaim
spec:
spec:
devices:
requests:
- name: request
firstAvailable:
- name: gpu
deviceClassName: resource.vendor.com
selectors:
- cel:
expression: |-
device.attributes["driver.vendor.com"].type == "gpu" &&
device.capacity["driver.vendor.com"].memory == quantity("80Gi")The NVIDIA DRA Driver is installed via a helm chart.
Let's dive into NVDIA's DRA Driver by first adding the NVIDIA helm repository:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo updateOutput:
"nvidia" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "nvidia" chart repository
Update Complete. ⎈Happy Helming!⎈Let's confirm the chart:
helm show chart nvidia/nvidia-dra-driver-gpuOutput:
apiVersion: v2
appVersion: 25.8.0
description: Official Helm chart for the NVIDIA DRA Driver for GPUs
kubeVersion: '>=1.32.0-0'
name: nvidia-dra-driver-gpu
type: application
version: 25.8.0Note: there will be a lot of output in the following steps, we will highlight key sections.
The steps are to help show how you can introspect into a helm chart for a DRA driver.
As we look into NVIDIA's DRA driver, we will see many references to ComputeDomain.
With the NVIDIA DRA driver, workloads request a ComputeDomain then NVIDIA's DRA Driver for GPUs works to share GPU memory securely with NVLink, a high-speed interconnect for GPUs and CPUs, 7x faster than PCIe gen 5.
This allows Kubernetes to use a rack of small NVIDIA GPUs into a "supercomputer" such as the GB200 NVL72.
Let's look at the configuration options:
helm show values nvidia/nvidia-dra-driver-gpuOutput:
# Copyright 2023 NVIDIA CORPORATION
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Default values for k8s-dra-driver-gpu.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# Specify the driver root on the host.
# If the NVIDIA GPU driver is managed using the NVIDIA GPU Driver Container,
# this is typically /run/nvidia/driver.
# For driver installed directly on a host, a value of `/` is used.
nvidiaDriverRoot: /
# Optional path to the nvidia-cdi-hook executable.
# If not specified, the default path inferred from the nvidia-container-toolkit library version will be used.
nvidiaCDIHookPath: ""
nameOverride: ""
fullnameOverride: ""
namespaceOverride: ""
selectorLabelsOverride: {}
gpuResourcesEnabledOverride: false
allowDefaultNamespace: false
imagePullSecrets: []
image:
repository: nvcr.io/nvidia/k8s-dra-driver-gpu
pullPolicy: IfNotPresent
# Note: an empty string is translated to the `appVersion` string from
# the Helm chart YAML (effectively implementing the default value to be
# the current version). Also note that a "v" is prefixed to the
# `appVersion` value.
tag: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
resources:
gpus:
enabled: true
computeDomains:
enabled: true
# Feature gates configuration following Kubernetes patterns
# Configure feature gates as key-value pairs (feature_name: true/false)
# Examples:
# featureGates:
# ExampleFeature: false # Project-specific alpha feature
# ContextualLogging: true # Kubernetes logging feature (enabled by default)
# LoggingAlphaOptions: false # Kubernetes logging alpha features
# LoggingBetaOptions: true # Kubernetes logging beta features
featureGates: {}
# Log verbosity for all components. Zero or greater, higher number means higher
# verbosity. Regardless of this setting, messages of type Error, Warning, and
# Info(level 0) are always logged. Can also be set for individual components via
# environment variable (that takes precedence), see
# https://github.com/NVIDIA/k8s-dra-driver-gpu/wiki/Troubleshooting#controlling-log-verbosity
#
# An (incomplete) representation of which types of messages to expect with
# increasing verbosity level:
#
# Level 0:
# - Configuration detail (during process startup)
# - Kubelet plugins:
# - Permanent errors during device Prepare() and Unprepare()
#
# Level 1:
# - CD controller:
# - Confirm cleanup of stale objects
# - k8s client-go: feature gates
# - Kubelet plugins:
# - Device (un)prepare confirmation, with resource claim UID
# - Workqueue reconciliation failures (noisy: mainly expected, retryable
# errors)
# - CD daemon:
# - explicit 'wait for nodes update'
#
# Level 2:
# - reflector.go informer state: "Caches populated"
# - Kubelet plugins:
# - Acknowledge when Unprepare is a noop
# - CD controller:
# - Added/updated API object callback confirmation
#
# Level 3:
# - reflector.go informer state: "Listing and watching"
#
# Level 6:
# - round_trippers.go output (API server request/response detail)
# - Kubelet plugins:
# - GRPC request/response detail
# - Checkpoint file update confirmation
# - CD daemon:
# - explicit 'IP set did not change'
#
# Level 7:
# - Kubelet plugins:
# - Health check
logVerbosity: "4"
# Webhook configuration
webhook:
enabled: false
replicas: 1
servicePort: 443
containerPort: 443
priorityClassName: "system-cluster-critical"
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: "100%"
podAnnotations: {}
podSecurityContext: {}
nodeSelector: {}
tolerations: []
affinity: {}
containers:
webhook:
securityContext:
privileged: false
resources: {}
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
name: ""
# failurePolicy defines how the API server should handle requests if the webhook call fails.
# Options:
# - Fail : reject the request if the webhook call fails either due to cert errors, timeout or if the service is unreachable.
# - Ignore : allow the request to continue if the webhook call fails.
failurePolicy: Fail
# TLS certificate configuration
tls:
# Certificate management mode: "cert-manager" or "secret"
# - "cert-manager": Use cert-manager to automatically generate and manage certificates
# - "secret": Use a user-provided secret containing tls.crt and tls.key
mode: "cert-manager"
certManager:
# Issuer type: "selfsigned", "clusterissuer", or "issuer"
issuerType: "selfsigned"
# Issuer name (required when issuerType is "clusterissuer" or "issuer")
issuerName: ""
# Additional DNS names for the certificate
dnsNames: []
secret:
# Name of the secret containing tls.crt and tls.key
name: ""
# Base64-encoded CA certificate bundle for validating the webhook's TLS certificate (base64 encoded)
# Required when using secret mode.
# Note: Only include intermediate CA certificates, not root CA certificates
caBundle: ""
controller:
priorityClassName: "system-node-critical"
podAnnotations: {}
podSecurityContext: {}
nodeSelector: {}
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
computeDomain:
securityContext: {}
env: []
resources: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
kubeletPlugin:
priorityClassName: "system-node-critical"
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: "100%"
podAnnotations: {}
podSecurityContext: {}
nodeSelector: {}
tolerations: []
kubeletRegistrarDirectoryPath: /var/lib/kubelet/plugins_registry
kubeletPluginsDirectoryPath: /var/lib/kubelet/plugins
containers:
init:
securityContext: {}
resources: {}
computeDomains:
env: []
securityContext:
privileged: true
resources: {}
# Port running a gRPC health service checked by a livenessProbe.
# Set to a negative value to disable the service and the probe.
healthcheckPort: 51515
gpus:
env: []
securityContext:
privileged: true
resources: {}
# Port running a gRPC health service checked by a livenessProbe.
# Set to a negative value to disable the service and the probe.
healthcheckPort: 51516
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
# On discrete-GPU based systems NFD adds the following label where 10de is the NVIDIA PCI vendor ID
- key: feature.node.kubernetes.io/pci-10de.present
operator: In
values:
- "true"
- matchExpressions:
# On some Tegra-based systems NFD detects the CPU vendor ID as NVIDIA
- key: feature.node.kubernetes.io/cpu-model.vendor_id
operator: In
values:
- "NVIDIA"
- matchExpressions:
# We allow a GPU deployment to be forced by setting the following label to "true"
- key: "nvidia.com/gpu.present"
operator: In
values:
- "true"As you can see, there are many options that can be configured during helm install.
For more information, see the NVIDIA docs.
Let's look at all the information from the chart (this will be a lot of output):
helm show all nvidia/nvidia-dra-driver-gpuOutput:
apiVersion: v2
appVersion: 25.8.0
description: Official Helm chart for the NVIDIA DRA Driver for GPUs
kubeVersion: '>=1.32.0-0'
name: nvidia-dra-driver-gpu
type: application
version: 25.8.0
---
# Copyright 2023 NVIDIA CORPORATION
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Default values for k8s-dra-driver-gpu.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# Specify the driver root on the host.
# If the NVIDIA GPU driver is managed using the NVIDIA GPU Driver Container,
# this is typically /run/nvidia/driver.
# For driver installed directly on a host, a value of `/` is used.
nvidiaDriverRoot: /
# Optional path to the nvidia-cdi-hook executable.
# If not specified, the default path inferred from the nvidia-container-toolkit library version will be used.
nvidiaCDIHookPath: ""
nameOverride: ""
fullnameOverride: ""
namespaceOverride: ""
selectorLabelsOverride: {}
gpuResourcesEnabledOverride: false
allowDefaultNamespace: false
imagePullSecrets: []
image:
repository: nvcr.io/nvidia/k8s-dra-driver-gpu
pullPolicy: IfNotPresent
# Note: an empty string is translated to the `appVersion` string from
# the Helm chart YAML (effectively implementing the default value to be
# the current version). Also note that a "v" is prefixed to the
# `appVersion` value.
tag: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
resources:
gpus:
enabled: true
computeDomains:
enabled: true
# Feature gates configuration following Kubernetes patterns
# Configure feature gates as key-value pairs (feature_name: true/false)
# Examples:
# featureGates:
# ExampleFeature: false # Project-specific alpha feature
# ContextualLogging: true # Kubernetes logging feature (enabled by default)
# LoggingAlphaOptions: false # Kubernetes logging alpha features
# LoggingBetaOptions: true # Kubernetes logging beta features
featureGates: {}
# Log verbosity for all components. Zero or greater, higher number means higher
# verbosity. Regardless of this setting, messages of type Error, Warning, and
# Info(level 0) are always logged. Can also be set for individual components via
# environment variable (that takes precedence), see
# https://github.com/NVIDIA/k8s-dra-driver-gpu/wiki/Troubleshooting#controlling-log-verbosity
#
# An (incomplete) representation of which types of messages to expect with
# increasing verbosity level:
#
# Level 0:
# - Configuration detail (during process startup)
# - Kubelet plugins:
# - Permanent errors during device Prepare() and Unprepare()
#
# Level 1:
# - CD controller:
# - Confirm cleanup of stale objects
# - k8s client-go: feature gates
# - Kubelet plugins:
# - Device (un)prepare confirmation, with resource claim UID
# - Workqueue reconciliation failures (noisy: mainly expected, retryable
# errors)
# - CD daemon:
# - explicit 'wait for nodes update'
#
# Level 2:
# - reflector.go informer state: "Caches populated"
# - Kubelet plugins:
# - Acknowledge when Unprepare is a noop
# - CD controller:
# - Added/updated API object callback confirmation
#
# Level 3:
# - reflector.go informer state: "Listing and watching"
#
# Level 6:
# - round_trippers.go output (API server request/response detail)
# - Kubelet plugins:
# - GRPC request/response detail
# - Checkpoint file update confirmation
# - CD daemon:
# - explicit 'IP set did not change'
#
# Level 7:
# - Kubelet plugins:
# - Health check
logVerbosity: "4"
# Webhook configuration
webhook:
enabled: false
replicas: 1
servicePort: 443
containerPort: 443
priorityClassName: "system-cluster-critical"
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: "100%"
podAnnotations: {}
podSecurityContext: {}
nodeSelector: {}
tolerations: []
affinity: {}
containers:
webhook:
securityContext:
privileged: false
resources: {}
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
name: ""
# failurePolicy defines how the API server should handle requests if the webhook call fails.
# Options:
# - Fail : reject the request if the webhook call fails either due to cert errors, timeout or if the service is unreachable.
# - Ignore : allow the request to continue if the webhook call fails.
failurePolicy: Fail
# TLS certificate configuration
tls:
# Certificate management mode: "cert-manager" or "secret"
# - "cert-manager": Use cert-manager to automatically generate and manage certificates
# - "secret": Use a user-provided secret containing tls.crt and tls.key
mode: "cert-manager"
certManager:
# Issuer type: "selfsigned", "clusterissuer", or "issuer"
issuerType: "selfsigned"
# Issuer name (required when issuerType is "clusterissuer" or "issuer")
issuerName: ""
# Additional DNS names for the certificate
dnsNames: []
secret:
# Name of the secret containing tls.crt and tls.key
name: ""
# Base64-encoded CA certificate bundle for validating the webhook's TLS certificate (base64 encoded)
# Required when using secret mode.
# Note: Only include intermediate CA certificates, not root CA certificates
caBundle: ""
controller:
priorityClassName: "system-node-critical"
podAnnotations: {}
podSecurityContext: {}
nodeSelector: {}
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
computeDomain:
securityContext: {}
env: []
resources: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
kubeletPlugin:
priorityClassName: "system-node-critical"
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: "100%"
podAnnotations: {}
podSecurityContext: {}
nodeSelector: {}
tolerations: []
kubeletRegistrarDirectoryPath: /var/lib/kubelet/plugins_registry
kubeletPluginsDirectoryPath: /var/lib/kubelet/plugins
containers:
init:
securityContext: {}
resources: {}
computeDomains:
env: []
securityContext:
privileged: true
resources: {}
# Port running a gRPC health service checked by a livenessProbe.
# Set to a negative value to disable the service and the probe.
healthcheckPort: 51515
gpus:
env: []
securityContext:
privileged: true
resources: {}
# Port running a gRPC health service checked by a livenessProbe.
# Set to a negative value to disable the service and the probe.
healthcheckPort: 51516
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
# On discrete-GPU based systems NFD adds the following label where 10de is the NVIDIA PCI vendor ID
- key: feature.node.kubernetes.io/pci-10de.present
operator: In
values:
- "true"
- matchExpressions:
# On some Tegra-based systems NFD detects the CPU vendor ID as NVIDIA
- key: feature.node.kubernetes.io/cpu-model.vendor_id
operator: In
values:
- "NVIDIA"
- matchExpressions:
# We allow a GPU deployment to be forced by setting the following label to "true"
- key: "nvidia.com/gpu.present"
operator: In
values:
- "true"
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.17.1
name: computedomains.resource.nvidia.com
spec:
group: resource.nvidia.com
names:
kind: ComputeDomain
listKind: ComputeDomainList
plural: computedomains
singular: computedomain
scope: Namespaced
versions:
- name: v1beta1
schema:
openAPIV3Schema:
description: ComputeDomain prepares a set of nodes to run a multi-node workload
in.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: ComputeDomainSpec provides the spec for a ComputeDomain.
properties:
channel:
description: ComputeDomainChannelSpec provides the spec for a channel
used to run a workload inside a ComputeDomain.
properties:
allocationMode:
default: Single
description: |-
Allows for requesting all IMEX channels (the maximum per IMEX domain) or
precisely one.
enum:
- All
- Single
type: string
resourceClaimTemplate:
description: ComputeDomainResourceClaimTemplate provides the details
of the ResourceClaimTemplate to generate.
properties:
name:
type: string
required:
- name
type: object
required:
- resourceClaimTemplate
type: object
numNodes:
description: |-
Intended number of IMEX daemons (i.e., individual compute nodes) in the
ComputeDomain. Must be zero or greater.
With `featureGates.IMEXDaemonsWithDNSNames=true` (the default), this is
recommended to be set to zero. Workload must implement and consult its
own source of truth for the number of workers online before trying to
share GPU memory (and hence triggering IMEX interaction). When non-zero,
`numNodes` is used only for automatically updating the global
ComputeDomain `Status` (indicating `Ready` when the number of ready IMEX
daemons equals `numNodes`). In this mode, a `numNodes` value greater than
zero in particular does not gate the startup of IMEX daemons: individual
IMEX daemons are started immediately without waiting for its peers, and
any workload pod gets released right after its local IMEX daemon has
started.
With `featureGates.IMEXDaemonsWithDNSNames=false`, `numNodes` must be set
to the expected number of worker nodes joining the ComputeDomain. In that
mode, all workload pods are held back (with containers in state
`ContainerCreating`) until the underlying IMEX domain has been joined by
`numNodes` IMEX daemons. Pods from more than `numNodes` nodes trying to
join the ComputeDomain may lead to unexpected behavior.
The `numNodes` parameter is deprecated and will be removed in the next
API version.
type: integer
required:
- channel
- numNodes
type: object
x-kubernetes-validations:
- message: A computeDomain.spec is immutable
rule: self == oldSelf
status:
description: |-
Global ComputeDomain status. Can be used to guide debugging efforts.
Workload however should not rely on inspecting this field at any point
during its lifecycle.
properties:
nodes:
items:
description: ComputeDomainNode provides information about each node
added to a ComputeDomain.
properties:
cliqueID:
type: string
index:
description: |-
The Index field is used to ensure a consistent IP-to-DNS name
mapping across all machines within an IMEX domain. Each node's index
directly determines its DNS name within a given NVLink partition
(i.e. clique). In other words, the 2-tuple of (CliqueID, Index) will
always be unique. This field is marked as optional (but not
omitempty) in order to support downgrades and avoid an API bump.
type: integer
ipAddress:
type: string
name:
type: string
status:
default: NotReady
description: |-
The Status field tracks the readiness of the IMEX daemon running on
this node. It gets switched to Ready whenever the IMEX daemon is
ready to broker GPU memory exchanges and switches to NotReady when
it is not. It is marked as optional in order to support downgrades
and avoid an API bump.
enum:
- Ready
- NotReady
type: string
required:
- cliqueID
- ipAddress
- name
type: object
type: array
x-kubernetes-list-map-keys:
- name
x-kubernetes-list-type: map
status:
default: NotReady
enum:
- Ready
- NotReady
type: string
required:
- status
type: object
type: object
served: true
storage: true
subresources:
status: {}
As we reviewed, the DRA driver is made up of 2 components: controller and a kubelet plugin. Let's take a look at the NVIDIA DRA driver's controller:
helm show all nvidia/nvidia-dra-driver-gpu | grep -A 21 "controller:" Output:
# - CD controller:
# - Confirm cleanup of stale objects
# - k8s client-go: feature gates
# - Kubelet plugins:
# - Device (un)prepare confirmation, with resource claim UID
# - Workqueue reconciliation failures (noisy: mainly expected, retryable
# errors)
# - CD daemon:
# - explicit 'wait for nodes update'
#
# Level 2:
# - reflector.go informer state: "Caches populated"
# - Kubelet plugins:
# - Acknowledge when Unprepare is a noop
# - CD controller:
# - Added/updated API object callback confirmation
#
# Level 3:
# - reflector.go informer state: "Listing and watching"
#
# Level 6:
# - round_trippers.go output (API server request/response detail)
# - Kubelet plugins:
# - GRPC request/response detail
# - Checkpoint file update confirmation
# - CD daemon:
# - explicit 'IP set did not change'
#
# Level 7:
# - Kubelet plugins:
# - Health check
logVerbosity: "4"
# Webhook configuration
webhook:
enabled: false
--
controller:
priorityClassName: "system-node-critical"
podAnnotations: {}
podSecurityContext: {}
nodeSelector: {}
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
computeDomain:
securityContext: {}
env: []
resources: {}
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"Now let's take a look at the NVIDIA DRA driver's kubelet plugin:
helm show all nvidia/nvidia-dra-driver-gpu | grep -A 19 "kubeletPlugin:" Output:
kubeletPlugin:
priorityClassName: "system-node-critical"
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: "100%"
podAnnotations: {}
podSecurityContext: {}
nodeSelector: {}
tolerations: []
kubeletRegistrarDirectoryPath: /var/lib/kubelet/plugins_registry
kubeletPluginsDirectoryPath: /var/lib/kubelet/plugins
containers:
init:
securityContext: {}
resources: {}
computeDomains:
env: []
securityContext:
privileged: trueTo install the NVIDIA DRA driver, we would do a helm install.
Let's do a dry-run to simulate the Helm chart install:
helm install --dry-run nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
--create-namespace \
--namespace nvidia-dra-driver-gpu \
--set resources.gpus.enabled=false \
--set nvidiaDriverRoot=/run/nvidia/driverThe command above uses an operator-provided GPU driver, for host-provided GPU drivers use the --set resources.gpus.enabled=false option.
Output:
NAME: nvidia-dra-driver-gpu
LAST DEPLOYED: Tue Nov 4 04:05:56 2025
NAMESPACE: nvidia-dra-driver-gpu
STATUS: pending-install
REVISION: 1
TEST SUITE: None
HOOKS:
MANIFEST:
---
# Source: nvidia-dra-driver-gpu/templates/rbac-compute-domain-daemon.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: compute-domain-daemon-service-account
namespace: nvidia-dra-driver-gpu
---
# Source: nvidia-dra-driver-gpu/templates/rbac-controller.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nvidia-dra-driver-gpu-service-account-controller
namespace: nvidia-dra-driver-gpu
labels:
helm.sh/chart: nvidia-dra-driver-gpu-25.8.0
app.kubernetes.io/version: "25.8.0"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nvidia-dra-driver-gpu
app.kubernetes.io/instance: nvidia-dra-driver-gpu
---
# Source: nvidia-dra-driver-gpu/templates/rbac-kubeletplugin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nvidia-dra-driver-gpu-service-account-kubeletplugin
namespace: nvidia-dra-driver-gpu
labels:
helm.sh/chart: nvidia-dra-driver-gpu-25.8.0
app.kubernetes.io/version: "25.8.0"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nvidia-dra-driver-gpu
app.kubernetes.io/instance: nvidia-dra-driver-gpu
---
# Source: nvidia-dra-driver-gpu/templates/rbac-compute-domain-daemon.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: compute-domain-daemon-role
rules:
- apiGroups: ["resource.nvidia.com"]
resources: ["computedomains", "computedomains/status"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
# Source: nvidia-dra-driver-gpu/templates/rbac-controller.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nvidia-dra-driver-gpu-clusterrole-controller
rules:
- apiGroups: ["resource.nvidia.com"]
resources: ["computedomains"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["resource.nvidia.com"]
resources: ["computedomains/status"]
verbs: ["update"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaimtemplates"]
verbs: ["get", "list", "watch", "create", "update", "delete"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
# Source: nvidia-dra-driver-gpu/templates/rbac-kubeletplugin.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nvidia-dra-driver-gpu-clusterrole-kubeletplugin
rules:
- apiGroups: ["resource.nvidia.com"]
resources: ["computedomains"]
verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims"]
verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceslices"]
verbs: ["get", "list", "watch", "create", "update", "delete"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
# Source: nvidia-dra-driver-gpu/templates/rbac-compute-domain-daemon.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: compute-domain-daemon-role-binding-nvidia-dra-driver-gpu
subjects:
- kind: ServiceAccount
name: compute-domain-daemon-service-account
namespace: nvidia-dra-driver-gpu
roleRef:
kind: ClusterRole
name: compute-domain-daemon-role
apiGroup: rbac.authorization.k8s.io
---
# Source: nvidia-dra-driver-gpu/templates/rbac-controller.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: nvidia-dra-driver-gpu-clusterrole-binding-controller-nvidia-dra-driver-gpu
subjects:
- kind: ServiceAccount
name: nvidia-dra-driver-gpu-service-account-controller
namespace: nvidia-dra-driver-gpu
roleRef:
kind: ClusterRole
name: nvidia-dra-driver-gpu-clusterrole-controller
apiGroup: rbac.authorization.k8s.io
---
# Source: nvidia-dra-driver-gpu/templates/rbac-kubeletplugin.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: nvidia-dra-driver-gpu-clusterrole-binding-kubeletplugin-nvidia-dra-driver-gpu
subjects:
- kind: ServiceAccount
name: nvidia-dra-driver-gpu-service-account-kubeletplugin
namespace: nvidia-dra-driver-gpu
roleRef:
kind: ClusterRole
name: nvidia-dra-driver-gpu-clusterrole-kubeletplugin
apiGroup: rbac.authorization.k8s.io
---
# Source: nvidia-dra-driver-gpu/templates/rbac-controller.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: nvidia-dra-driver-gpu-role-controller
namespace: nvidia-dra-driver-gpu
rules:
- apiGroups: ["apps"]
resources: ["daemonsets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
# Source: nvidia-dra-driver-gpu/templates/rbac-kubeletplugin.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: nvidia-dra-driver-gpu-role-kubeletplugin
namespace: nvidia-dra-driver-gpu
rules:
---
# Source: nvidia-dra-driver-gpu/templates/rbac-controller.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: nvidia-dra-driver-gpu-role-binding-controller
namespace: nvidia-dra-driver-gpu
subjects:
- kind: ServiceAccount
name: nvidia-dra-driver-gpu-service-account-controller
namespace: nvidia-dra-driver-gpu
roleRef:
kind: Role
name: nvidia-dra-driver-gpu-role-controller
apiGroup: rbac.authorization.k8s.io
---
# Source: nvidia-dra-driver-gpu/templates/rbac-kubeletplugin.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: nvidia-dra-driver-gpu-role-binding-kubeletplugin
namespace: nvidia-dra-driver-gpu
subjects:
- kind: ServiceAccount
name: nvidia-dra-driver-gpu-service-account-kubeletplugin
namespace: nvidia-dra-driver-gpu
roleRef:
kind: Role
name: nvidia-dra-driver-gpu-role-kubeletplugin
apiGroup: rbac.authorization.k8s.io
---
# Source: nvidia-dra-driver-gpu/templates/kubeletplugin.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nvidia-dra-driver-gpu-kubelet-plugin
namespace: nvidia-dra-driver-gpu
labels:
helm.sh/chart: nvidia-dra-driver-gpu-25.8.0
app.kubernetes.io/version: "25.8.0"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nvidia-dra-driver-gpu
app.kubernetes.io/instance: nvidia-dra-driver-gpu
spec:
selector:
matchLabels:
nvidia-dra-driver-gpu-component: kubelet-plugin
updateStrategy:
rollingUpdate:
maxUnavailable: 100%
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: nvidia-dra-driver-gpu
app.kubernetes.io/instance: nvidia-dra-driver-gpu
nvidia-dra-driver-gpu-component: kubelet-plugin
spec:
priorityClassName: system-node-critical
serviceAccountName: nvidia-dra-driver-gpu-service-account-kubeletplugin
securityContext:
{}
initContainers:
- name: init-container
image: nvcr.io/nvidia/k8s-dra-driver-gpu:v25.8.0
securityContext:
privileged: true
command: [bash, /usr/bin/kubelet-plugin-prestart.sh]
env:
- name: NVIDIA_DRIVER_ROOT
value: "/run/nvidia/driver"
# Use runc: explicit "void"; otherwise we inherit "all".
- name: NVIDIA_VISIBLE_DEVICES
value: void
- name: KUBELET_REGISTRAR_DIRECTORY_PATH
value: "/var/lib/kubelet/plugins_registry"
- name: KUBELET_PLUGINS_DIRECTORY_PATH
value: "/var/lib/kubelet/plugins"
volumeMounts:
- name: driver-root-parent
mountPath: /driver-root-parent
# In case of the operator-provided driver, another container mounts
# the driver onto the host using `mountPropagation: Bidirectional`
# (out-of-band of the lifecycle of _this_ pod here). For us to see
# that mount, `mountPropagation: HostToContainer` is required (docs:
# "if any Pod with Bidirectional mount propagation to the same volume
# mounts anything there, the container with HostToContainer mount
# propagation will see it.").
mountPropagation: HostToContainer
containers:
- name: compute-domains
securityContext:
privileged: true
image: nvcr.io/nvidia/k8s-dra-driver-gpu:v25.8.0
imagePullPolicy: IfNotPresent
command: ["bash", "-c"]
args:
- |-
# Conditionally mask the params file to prevent this container from
# recreating any missing GPU device nodes. This is necessary, for
# example, when running under nvkind to limit the set GPUs governed
# by the plugin even though it has cgroup access to all of them.
if [ "${MASK_NVIDIA_DRIVER_PARAMS}" = "true" ]; then
cp /proc/driver/nvidia/params root/gpu-params
sed -i 's/^ModifyDeviceFiles: 1$/ModifyDeviceFiles: 0/' root/gpu-params
mount --bind root/gpu-params /proc/driver/nvidia/params
fi
compute-domain-kubelet-plugin -v $(LOG_VERBOSITY)
resources:
{}
livenessProbe:
grpc:
port: 51515
service: liveness
failureThreshold: 3
periodSeconds: 10
env:
# LOG_VERBOSITY is the source of truth for this program's klog
# configuration. Currently injected via CLI argument (see above) because
# klog's verbosity for now cannot be sanely set from an environment
# variable.
- name: LOG_VERBOSITY
value: "4"
- name: MASK_NVIDIA_DRIVER_PARAMS
value: ""
- name: NVIDIA_DRIVER_ROOT
value: "/run/nvidia/driver"
- name: NVIDIA_VISIBLE_DEVICES
value: void
- name: CDI_ROOT
value: /var/run/cdi
- name: NVIDIA_MIG_CONFIG_DEVICES
value: all
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBELET_REGISTRAR_DIRECTORY_PATH
value: "/var/lib/kubelet/plugins_registry"
- name: KUBELET_PLUGINS_DIRECTORY_PATH
value: "/var/lib/kubelet/plugins"
- name: HEALTHCHECK_PORT
value: "51515"
volumeMounts:
- name: plugins-registry
mountPath: "/var/lib/kubelet/plugins_registry"
- name: plugins
mountPath: "/var/lib/kubelet/plugins"
mountPropagation: Bidirectional
- name: cdi
mountPath: /var/run/cdi
- name: driver-root
mountPath: /driver-root
readOnly: true
mountPropagation: HostToContainer
volumes:
- name: plugins-registry
hostPath:
path: "/var/lib/kubelet/plugins_registry"
- name: plugins
hostPath:
path: "/var/lib/kubelet/plugins"
- name: cdi
hostPath:
path: /var/run/cdi
- name: driver-root-parent
hostPath:
# If nvidiaDriverRoot == "/" then its parent is itself. Otherwise, get
# its parent by removing any trailing slashes as well as the last path
# element with sprig template function `dir`. Examples: /a/b/ -> /a,
# /a/b/c -> /a/b.
path: /run/nvidia
type: DirectoryOrCreate
- name: driver-root
hostPath:
path: /run/nvidia/driver
type: DirectoryOrCreate
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: feature.node.kubernetes.io/pci-10de.present
operator: In
values:
- "true"
- matchExpressions:
- key: feature.node.kubernetes.io/cpu-model.vendor_id
operator: In
values:
- NVIDIA
- matchExpressions:
- key: nvidia.com/gpu.present
operator: In
values:
- "true"
---
# Source: nvidia-dra-driver-gpu/templates/controller.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nvidia-dra-driver-gpu-controller
namespace: nvidia-dra-driver-gpu
labels:
helm.sh/chart: nvidia-dra-driver-gpu-25.8.0
app.kubernetes.io/version: "25.8.0"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nvidia-dra-driver-gpu
app.kubernetes.io/instance: nvidia-dra-driver-gpu
spec:
replicas: 1
selector:
matchLabels:
nvidia-dra-driver-gpu-component: controller
template:
metadata:
labels:
app.kubernetes.io/name: nvidia-dra-driver-gpu
app.kubernetes.io/instance: nvidia-dra-driver-gpu
nvidia-dra-driver-gpu-component: controller
spec:
priorityClassName: system-node-critical
serviceAccountName: nvidia-dra-driver-gpu-service-account-controller
securityContext:
{}
containers:
- name: compute-domain
securityContext:
{}
image: nvcr.io/nvidia/k8s-dra-driver-gpu:v25.8.0
imagePullPolicy: IfNotPresent
command: ["compute-domain-controller", "-v", "$(LOG_VERBOSITY)"]
resources:
{}
env:
# LOG_VERBOSITY is the source of truth for this program's klog
# configuration. Currently injected via CLI argument (see above) because
# klog's verbosity for now cannot be sanely set from an env var.
- name: LOG_VERBOSITY
value: "4"
# LOG_VERBOSITY_CD_DAEMON controls the verbosity of dynamically launched
# CD daemons (their pod spec is not rendered by Helm, but by this
# controller).
- name: LOG_VERBOSITY_CD_DAEMON
value: "4"
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: IMAGE_NAME
value: nvcr.io/nvidia/k8s-dra-driver-gpu:v25.8.0
# Use runc: explicit "void"; otherwise we inherit "all".
- name: NVIDIA_VISIBLE_DEVICES
value: void
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
---
# Source: nvidia-dra-driver-gpu/templates/controller.yaml
# Copyright 2024 NVIDIA CORPORATION
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
# Source: nvidia-dra-driver-gpu/templates/kubeletplugin.yaml
# Copyright 2023 NVIDIA CORPORATION
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
# Source: nvidia-dra-driver-gpu/templates/openshiftprivilegedrolebinging.yaml
# Apply only when running on OpenShift to let the kublet plugin run privileged
---
# Source: nvidia-dra-driver-gpu/templates/validation.yaml
# Copyright 2024 NVIDIA CORPORATION
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
# Source: nvidia-dra-driver-gpu/templates/webhook-deployment.yaml
# Copyright 2025 NVIDIA CORPORATION
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
# Source: nvidia-dra-driver-gpu/templates/deviceclass-compute-domain-daemon.yaml
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: compute-domain-daemon.nvidia.com
spec:
selectors:
- cel:
expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'daemon'"
---
# Source: nvidia-dra-driver-gpu/templates/deviceclass-compute-domain-default-channel.yaml
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: compute-domain-default-channel.nvidia.com
spec:
selectors:
- cel:
expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'channel' && device.attributes['compute-domain.nvidia.com'].id == 0"
---
# Source: nvidia-dra-driver-gpu/templates/validatingadmissionpolicy.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: resourceslices-policy-nvidia-dra-driver-gpu
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["resource.k8s.io"]
apiVersions: ["v1", "v1beta1", "v1beta2"]
operations: ["CREATE", "UPDATE", "DELETE"]
resources: ["resourceslices"]
matchConditions:
- name: isRestrictedUser
expression: >-
request.userInfo.username == "system:serviceaccount:nvidia-dra-driver-gpu:nvidia-dra-driver-gpu-service-account"
variables:
- name: userNodeName
expression: >-
request.userInfo.extra[?'authentication.kubernetes.io/node-name'][0].orValue('')
- name: objectNodeName
expression: >-
(request.operation == "DELETE" ? oldObject : object).spec.?nodeName.orValue("")
- name: nodeSelectorValue
expression: >-
(request.operation == "DELETE" ? oldObject : object).spec.?nodeSelector.orValue(null)
- name: allNodesValue
expression: >-
(request.operation == "DELETE" ? oldObject : object).spec.?allNodes.orValue(false)
validations:
- expression: variables.userNodeName != ""
message: >-
no node association found for user, this user must run in a pod on a node and ServiceAccountTokenPodNodeInfo must be enabled
- expression: variables.userNodeName == variables.objectNodeName || variables.allNodesValue == true || variables.nodeSelectorValue != null
messageExpression: >-
"this user running on node '"+variables.userNodeName+"' may not modify cluster or node resourceslices"
---
# Source: nvidia-dra-driver-gpu/templates/validatingadmissionpolicybinding.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: resourceslices-policy-nvidia-dra-driver-gpu
spec:
policyName: resourceslices-policy-nvidia-dra-driver-gpu
validationActions: [Deny]
# All ResourceSlices are matched.The NVIDIA DRA helm chart installed the following:
- Namespace: nvidia-dra-driver-gpu
- ServiceAccount: compute-domain-daemon-service-account
- ServiceAccount: nvidia-dra-driver-gpu-service-account-controller
- ServiceAccount: nvidia-dra-driver-gpu-service-account-kubeletplugin
- ClusterRole: compute-domain-daemon-role
- ClusterRole: nvidia-dra-driver-gpu-clusterrole-controller
- ClusterRole: nvidia-dra-driver-gpu-clusterrole-kubeletplugin
- ClusterRoleBinding: compute-domain-daemon-role-binding-nvidia-dra-driver-gpu
- binds ServiceAccount compute-domain-daemon-service-account and ClusterRole compute-domain-daemon-role
- ClusterRoleBinding: nvidia-dra-driver-gpu-clusterrole-binding-controller-nvidia-dra-driver-gpu
- binds ServiceAccount nvidia-dra-driver-gpu-service-account-controller and ClusterRole nvidia-dra-driver-gpu-clusterrole-controller
- ClusterRoleBinding: nvidia-dra-driver-gpu-clusterrole-binding-kubeletplugin-nvidia-dra-driver-gpu
- binds ServiceAccount nvidia-dra-driver-gpu-service-account-kubeletplugin and ClusterRole nvidia-dra-driver-gpu-clusterrole-kubeletplugin
- Role: nvidia-dra-driver-gpu-role-controller
- Role: nvidia-dra-driver-gpu-role-kubeletplugin
- RoleBinding: nvidia-dra-driver-gpu-role-binding-controller
- binds ServiceAccount nvidia-dra-driver-gpu-service-account-controller and Role nvidia-dra-driver-gpu-role-controller
- RoleBinding: nvidia-dra-driver-gpu-role-binding-kubeletplugin
- binds ServiceAccount nvidia-dra-driver-gpu-service-account-kubeletplugin and Role nvidia-dra-driver-gpu-role-kubeletplugin
- DaemonSet: nvidia-dra-driver-gpu-kubelet-plugin
- Deployment: nvidia-dra-driver-gpu-controller
- DeviceClass: compute-domain-daemon.nvidia.com
- DeviceClass: compute-domain-default-channel.nvidia.com
- ValidatingAdmissionPolicy: resourceslices-policy-nvidia-dra-driver-gpu
- ValidatingAdmissionPolicyBinding: resourceslices-policy-nvidia-dra-driver-gpu
For more information on the NVIDIA DRA driver installation see https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-intro-install.html.
Intel hosts its GPU DRA Driver on GitHub intel/intel-resource-drivers-for-kubernetes.
Intel's GPU DRA Driver is also installed via a Helm chart.
Let's look at Intel's GPU DRA Driver through a dry-run simulation of the helm install:
helm install --dry-run intel-gpu-resource-driver oci://ghcr.io/intel/intel-resource-drivers-for-kubernetes/intel-gpu-resource-driver-chart \
--namespace "intel-gpu-resource-driver" \
--create-namespace Output:
Pulled: ghcr.io/intel/intel-resource-drivers-for-kubernetes/intel-gpu-resource-driver-chart:0.9.0
Digest: sha256:2c1945239fbc8c060460428d3f2c71dd4cec2136e68a83e4fada5ff7b21dfb34
NAME: intel-gpu-resource-driver
LAST DEPLOYED: Tue Nov 4 04:13:28 2025
NAMESPACE: intel-gpu-resource-driver
STATUS: pending-install
REVISION: 1
TEST SUITE: None
HOOKS:
MANIFEST:
---
# Source: intel-gpu-resource-driver-chart/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: intel-gpu-sa
namespace: intel-gpu-resource-driver
labels:
helm.sh/chart: intel-gpu-resource-driver-chart-0.9.0
app.kubernetes.io/version: "v0.9.0"
app.kubernetes.io/managed-by: Helm
automountServiceAccountToken: true
---
# Source: intel-gpu-resource-driver-chart/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: intel-gpu-resource-driver-role
namespace: intel-gpu-resource-driver
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceslices"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims"]
verbs: ["get"]
---
# Source: intel-gpu-resource-driver-chart/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: intel-gpu-resource-driver-rolebinding
namespace: intel-gpu-resource-driver
subjects:
- kind: ServiceAccount
name: intel-gpu-sa
namespace: intel-gpu-resource-driver
roleRef:
kind: ClusterRole
name: intel-gpu-resource-driver-role
apiGroup: rbac.authorization.k8s.io
---
# Source: intel-gpu-resource-driver-chart/templates/resource-driver.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-gpu-resource-driver-kubelet-plugin
namespace: intel-gpu-resource-driver
labels:
helm.sh/chart: intel-gpu-resource-driver-chart-0.9.0
app.kubernetes.io/version: "v0.9.0"
app.kubernetes.io/managed-by: Helm
spec:
selector:
matchLabels:
app: intel-gpu-resource-driver
template:
metadata:
labels:
app: intel-gpu-resource-driver
spec:
serviceAccountName: intel-gpu-sa
containers:
- name: kubelet-plugin
image: ghcr.io/intel/intel-resource-drivers-for-kubernetes/intel-gpu-resource-driver:v0.9.0
imagePullPolicy: IfNotPresent
command: ["/kubelet-gpu-plugin"]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: SYSFS_ROOT
value: "/sysfs"
- name: ZES_ENABLE_SYSMAN
value: "1"
volumeMounts:
- name: plugins-registry
mountPath: /var/lib/kubelet/plugins_registry
- name: plugins
mountPath: /var/lib/kubelet/plugins
- name: cdi
mountPath: /etc/cdi
- name: varruncdi
mountPath: /var/run/cdi
# when using fake sysfs - mount at the same place as on host
- name: sysfs
mountPath: "/sysfs"
securityContext:
privileged: true
capabilities:
drop: ["ALL"]
readOnlyRootFilesystem: true
runAsUser: 0
seccompProfile:
type: RuntimeDefault
volumes:
- name: plugins-registry
hostPath:
path: /var/lib/kubelet/plugins_registry
- name: plugins
hostPath:
path: /var/lib/kubelet/plugins
- name: cdi
hostPath:
path: /etc/cdi
- name: varruncdi
hostPath:
path: /var/run/cdi
- name: sysfs
hostPath:
path: /sys
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/gpu
operator: Exists
---
# Source: intel-gpu-resource-driver-chart/templates/device-class.yaml
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: gpu.intel.com
spec:
selectors:
- cel:
expression: device.driver == "gpu.intel.com"
# Available in K8s v1.34 requires feature gate enabled
# See https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5004-dra-extended-resource
extendedResourceName: intel.com/gpu
---
# Source: intel-gpu-resource-driver-chart/templates/validating-admission-policy.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: resourceslices-policy-dra-kubelet-plugin-gpu
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["resource.k8s.io"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE", "DELETE"]
resources: ["resourceslices"]
matchConditions:
- name: isRestrictedUser
expression: >-
request.userInfo.username == "system:serviceaccount:intel-gpu-resource-driver:intel-gpu-sa"
variables:
- name: userNodeName
expression: >-
request.userInfo.extra[?'authentication.kubernetes.io/node-name'][0].orValue('')
- name: objectNodeName
expression: >-
(request.operation == "DELETE" ? oldObject : object).spec.?nodeName.orValue("")
validations:
- expression: variables.userNodeName != ""
message: >-
no node association found for user, this user must run in a pod on a node and ServiceAccountTokenPodNodeInfo must be enabled
- expression: variables.userNodeName == variables.objectNodeName
messageExpression: >-
"this user running on node '"+variables.userNodeName+"' may not modify " +
(variables.objectNodeName == "" ?"cluster resourceslices" : "resourceslices on node '"+variables.objectNodeName+"'")
---
# Source: intel-gpu-resource-driver-chart/templates/validating-admission-policy-binding.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: resourceslices-policy-dra-kubelet-plugin-gpu
spec:
policyName: resourceslices-policy-dra-kubelet-plugin-gpu
validationActions: [Deny]
NOTES:
Thank you for installing intel-gpu-resource-driver-chart.The Intel DRA helm chart installed the following:
- Namespace: intel-gpu-resource-driver
- ServiceAccount: intel-gpu-sa
- ClusterRole: intel-gpu-resource-driver-role
- ClusterRoleBinding: intel-gpu-resource-driver-rolebinding
- binds ServiceAccount intel-gpu-sa and ClusterRole intel-gpu-resource-driver-role
- DaemonSet: intel-gpu-resource-driver-kubelet-plugin
- DeviceClass: gpu.intel.com
- ValidatingAdmissionPolicy: resourceslices-policy-dra-kubelet-plugin-gpu
- ValidatingAdmissionPolicyBinding: resourceslices-policy-dra-kubelet-plugin-gpu
Since our environment does not have real GPUs, in the next module we will install example GPU drivers.
Dynamic Resource Allocation is not specific to GPUs.
Google's DRANET is a Kubernetes Network Driver for high-performance networking.
The DRANET driver communicated with the kubelet though the DRA API and the container runtime via the node resource interface.
Then the Pod's network namespace is created, the container runtime initiates a GRPC call to DRANET via node resource interface.
More on DRANET can be found here:
https://github.com/google/dranet
https://dranet.dev/docs/
DRANET is installed via a yaml file.
Let's inspect the yaml file:
curl https://raw.githubusercontent.com/google/dranet/refs/heads/main/install.yamlOutput:
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: dranet
rules:
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- "resource.k8s.io"
resources:
- resourceslices
verbs:
- list
- watch
- create
- update
- delete
- apiGroups:
- "resource.k8s.io"
resources:
- resourceclaims
- deviceclasses
verbs:
- get
- apiGroups:
- "resource.k8s.io"
resources:
- resourceclaims/status
verbs:
- patch
- update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: dranet
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dranet
subjects:
- kind: ServiceAccount
name: dranet
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: dranet
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dranet
namespace: kube-system
labels:
tier: node
app: dranet
k8s-app: dranet
spec:
selector:
matchLabels:
app: dranet
template:
metadata:
labels:
tier: node
app: dranet
k8s-app: dranet
spec:
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: dranet
hostPID: true
initContainers:
- name: enable-nri
image: busybox:stable
volumeMounts:
- mountPath: /etc
name: etc
securityContext:
privileged: true
command:
- /bin/sh
- -c
- |
set -o errexit
set -o pipefail
set -o nounset
set -x
if grep -q "io.containerd.nri.v1.nri" /etc/containerd/config.toml
then
echo "containerd config contains NRI reference already; taking no action"
else
echo "containerd config does not mention NRI, thus enabling it";
printf '%s\n' "[plugins.\"io.containerd.nri.v1.nri\"]" " disable = false" " disable_connections = false" " plugin_config_path = \"/etc/nri/conf.d\"" " plugin_path = \"/opt/nri/plugins\"" " plugin_registration_timeout = \"5s\"" " plugin_request_timeout = \"5s\"" " socket_path = \"/var/run/nri/nri.sock\"" >> /etc/containerd/config.toml
echo "restarting containerd"
nsenter -t 1 -m -u -i -n -p -- systemctl restart containerd
fi
containers:
- name: dranet
args:
- /dranet
- --v=4
- --hostname-override=$(NODE_NAME)
image: ghcr.io/google/dranet:stable
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: true
readinessProbe:
httpGet:
path: /healthz
port: 9177
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/plugins
- name: plugin-registry
mountPath: /var/lib/kubelet/plugins_registry
- name: nri-plugin
mountPath: /var/run/nri
- name: netns
mountPath: /var/run/netns
mountPropagation: HostToContainer
- name: infiniband
mountPath: /dev/infiniband
mountPropagation: HostToContainer
- name: bpf-programs
mountPath: /sys/fs/bpf
mountPropagation: HostToContainer
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/plugins
- name: plugin-registry
hostPath:
path: /var/lib/kubelet/plugins_registry
- name: nri-plugin
hostPath:
path: /var/run/nri
- name: netns
hostPath:
path: /var/run/netns
- name: infiniband
hostPath:
path: /dev/infiniband
- name: etc
hostPath:
path: /etc
- name: bpf-programs
hostPath:
path: /sys/fs/bpf
---Let's take a look at the objects that would be created with a --dry-run=client.
kubectl apply -f https://raw.githubusercontent.com/google/dranet/refs/heads/main/install.yaml --dry-run=clientOutput:
clusterrole.rbac.authorization.k8s.io/dranet created (dry run)
clusterrolebinding.rbac.authorization.k8s.io/dranet created (dry run)
serviceaccount/dranet created (dry run)
daemonset.apps/dranet created (dry run)A ResourceSlice would be created similar to the following:
apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
creationTimestamp: "2024-12-15T23:41:51Z"
generateName: gke-aojea-dra-multi-nic-985b8c20-jg5l-dra.net-
generation: 1
name: gke-aojea-dra-multi-nic-985b8c20-jg5l-dra.net-8nq9c
ownerReferences:
- apiVersion: v1
controller: true
kind: Node
name: gke-aojea-dra-multi-nic-985b8c20-jg5l
uid: 0146a07e-df67-401d-b3a5-dddb02f50b6e
resourceVersion: "1471803"
uid: 535724d7-a573-49e1-8f3b-4e644405375a
spec:
devices:
- basic:
attributes:
dra.net/alias:
string: ""
dra.net/cloudNetwork:
string: dra-1-vpc
dra.net/encapsulation:
string: ether
dra.net/ifName:
string: gpu7rdma0
dra.net/ipv4:
string: 10.0.8.8
dra.net/mac:
string: 9a:41:2e:4f:86:16
dra.net/mtu:
int: 8896
dra.net/numaNode:
int: 1
dra.net/pciAddressBus:
string: c8
dra.net/pciAddressDevice:
string: "00"
dra.net/pciAddressDomain:
string: "0000"
dra.net/pciAddressFunction:
string: "0"
dra.net/pciVendor:
string: Mellanox Technologies
dra.net/rdma:
bool: true
dra.net/sriov:
bool: false
dra.net/state:
string: up
dra.net/type:
string: device
dra.net/virtual:
bool: false
name: gpu7rdma0To use DRANET, create the DeviceClass and ResourceClaim (from the DRANET docs):
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: dranet-cloud
spec:
selectors:
- cel:
expression: device.driver == "dra.net"
- cel:
expression: has(device.attributes["dra.net"].cloudNetwork)
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
name: cloud-network-dra-net
spec:
devices:
requests:
- name: request-cloud-net
exactly:
deviceClassName: dranet-cloud
selectors:
- cel:
expression: device.attributes["dra.net"].cloudNetwork == "dra-1-vpc"An example Pod that uses DRANET looks like:
apiVersion: v1
kind: Pod
metadata:
name: pod-dra-net
labels:
app: pod-dra-net
spec:
containers:
- name: ctr1
image: registry.k8s.io/e2e-test-images/agnhost:2.39
resourceClaims:
- name: net-1
resourceClaimName: cloud-network-dra-netCreate the dra-tutorial namespace:
kubectl create namespace dra-tutorialOutput:
namespace/dra-tutorial createdCreate the DeviceClass that represents the supported devices of the DRA driver.
Let's take a look at the manifest for the DeviceClass:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/deviceclass.yamlOutput:
apiVersion: resource.k8s.io/v1beta1
kind: DeviceClass
metadata:
name: gpu.example.com
spec:
selectors:
- cel:
expression: "device.driver == 'gpu.example.com'"Commen Express Language (CEL) can be used to filter for specific attributes.
Let's compare this DeviceClass with the ones from NVIDIA and Intel:
NVIDIA:
helm install --dry-run nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
--create-namespace \
--namespace nvidia-dra-driver-gpu \
--set resources.gpus.enabled=false \
--set nvidiaDriverRoot=/run/nvidia/driver \
| grep -A 8 -B 2 "kind: DeviceClass"Output:
# Source: nvidia-dra-driver-gpu/templates/deviceclass-compute-domain-daemon.yaml
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: compute-domain-daemon.nvidia.com
spec:
selectors:
- cel:
expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'daemon'"
---
# Source: nvidia-dra-driver-gpu/templates/deviceclass-compute-domain-default-channel.yaml
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: compute-domain-default-channel.nvidia.com
spec:
selectors:
- cel:
expression: "device.driver == 'compute-domain.nvidia.com' && device.attributes['compute-domain.nvidia.com'].type == 'channel' && device.attributes['compute-domain.nvidia.com'].id == 0"
---
# Source: nvidia-dra-driver-gpu/templates/validatingadmissionpolicy.yamlIntel:
helm install --dry-run intel-gpu-resource-driver oci://ghcr.io/intel/intel-resource-drivers-for-kubernetes/intel-gpu-resource-driver-chart \
--namespace "intel-gpu-resource-driver" \
--create-namespace \
| grep -A 11 -B 2 "kind: DeviceClass"Output:
Pulled: ghcr.io/intel/intel-resource-drivers-for-kubernetes/intel-gpu-resource-driver-chart:0.9.0
Digest: sha256:2c1945239fbc8c060460428d3f2c71dd4cec2136e68a83e4fada5ff7b21dfb34
# Source: intel-gpu-resource-driver-chart/templates/device-class.yaml
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: gpu.intel.com
spec:
selectors:
- cel:
expression: device.driver == "gpu.intel.com"
# Available in K8s v1.34 requires feature gate enabled
# See https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5004-dra-extended-resource
extendedResourceName: intel.com/gpu
---Create the DeviceClass for the example DRA driver:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/deviceclass.yamlOutput:
deviceclass.resource.k8s.io/gpu.example.com createdBefore you deploy a DRA driver, create RBAC authorization for the DRA driver to control ResourceSlices, get Nodes, and get ResourceClaims.
Let's take a look at the ServiceAccount, ClusterRole, ClusterRoleBinding that binds the ClusterRole to the Service Account:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/rbac.yamlOutput:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: dra-example-driver-service-account
namespace: dra-tutorial
labels:
app.kubernetes.io/name: dra-example-driver
app.kubernetes.io/instance: dra-example-driver
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: dra-example-driver-role
rules:
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims"]
verbs: ["get"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceslices"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dra-example-driver-role-binding
subjects:
- kind: ServiceAccount
name: dra-example-driver-service-account
namespace: dra-tutorial
roleRef:
kind: ClusterRole
name: dra-example-driver-role
apiGroup: rbac.authorization.k8s.io
---Create the ServiceAccount, ClusterRole, and ClusteRoleBinding:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/rbac.yamlOutput:
serviceaccount/dra-example-driver-service-account created
clusterrole.rbac.authorization.k8s.io/dra-example-driver-role created
clusterrolebinding.rbac.authorization.k8s.io/dra-example-driver-role-binding createdCreate the PriorityClass to prevent preemption of the DRA driver:
Let's look at the PriorityClass:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/priorityclass.yamlOutput:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: dra-driver-high-priority
value: 1000000
globalDefault: false
description: "PriorityClass for DRA driver pods only to prevent preemption of the DRA driver."Create the PriorityClass:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/priorityclass.yamlOutput:
priorityclass.scheduling.k8s.io/dra-driver-high-priority createdBefore creating the example DRA driver, let's take a look at the DRA driver's manifest:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/dra-driver-daemonset.yaml Output:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dra-example-driver-kubeletplugin
namespace: dra-tutorial
labels:
app.kubernetes.io/name: dra-example-driver
spec:
selector:
matchLabels:
app.kubernetes.io/name: dra-example-driver
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: dra-example-driver
spec:
priorityClassName: dra-driver-high-priority
serviceAccountName: dra-example-driver-service-account
securityContext:
{}
containers:
- name: plugin
securityContext:
privileged: true
image: registry.k8s.io/dra-example-driver/dra-example-driver:v0.2.0
imagePullPolicy: IfNotPresent
command: ["dra-example-kubeletplugin"]
resources:
{}
# Production drivers should always implement a liveness probe
# For the tutorial we simply omit it
# livenessProbe:
# grpc:
# port: 51515
# service: liveness
# failureThreshold: 3
# periodSeconds: 10
env:
- name: CDI_ROOT
value: /var/run/cdi
- name: KUBELET_REGISTRAR_DIRECTORY_PATH
value: "/var/lib/kubelet/plugins_registry"
- name: KUBELET_PLUGINS_DIRECTORY_PATH
value: "/var/lib/kubelet/plugins"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# Simulated number of devices the example driver will pretend to have.
- name: NUM_DEVICES
value: "9"
- name: HEALTHCHECK_PORT
value: "51515"
volumeMounts:
- name: plugins-registry
mountPath: "/var/lib/kubelet/plugins_registry"
- name: plugins
mountPath: "/var/lib/kubelet/plugins"
- name: cdi
mountPath: /var/run/cdi
volumes:
- name: plugins-registry
hostPath:
path: "/var/lib/kubelet/plugins_registry"
- name: plugins
hostPath:
path: "/var/lib/kubelet/plugins"
- name: cdi
hostPath:
path: /var/run/cdiCreate the DRA driver in a DaemonSet, the driver binary is in a container image:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/dra-driver-daemonset.yamlOutput:
daemonset.apps/dra-example-driver-kubeletplugin createdCheck the status of the DRA driver:
kubectl get daemonset,pods -n dra-tutorial -l app.kubernetes.io/name=dra-example-driverOutput:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/dra-example-driver-kubeletplugin 1 1 1 1 1 <none> 2m17s
NAME READY STATUS RESTARTS AGE
pod/dra-example-driver-kubeletplugin-q7whd 1/1 Running 0 2m17sA ResourceSlice would be created. Let's take a look into the ResourceSlice.
kubectl get resourcesliceOutput:
NAME NODE DRIVER POOL AGE
kind-worker-gpu.example.com-w9pv9 kind-worker gpu.example.com kind-worker 7m48sDescribe the ResourceSlice to see the "fictional" devices:
kubectl describe $(kubectl get resourceslice -o name)Output:
Name: kind-worker-gpu.example.com-w9pv9
Namespace:
Labels: <none>
Annotations: <none>
API Version: resource.k8s.io/v1
Kind: ResourceSlice
Metadata:
Creation Timestamp: 2025-11-10T18:31:45Z
Generate Name: kind-worker-gpu.example.com-
Generation: 1
Owner References:
API Version: v1
Controller: true
Kind: Node
Name: kind-worker
UID: 939276b2-7940-4e27-a644-c53313bf4ad2
Resource Version: 16424
UID: 076766a8-4df1-40fa-ba05-bd345acf8772
Spec:
Devices:
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 7
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-f270be4e-7cd6-da75-39e7-b707122f9b70
Capacity:
Memory:
Value: 80Gi
Name: gpu-7
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 2
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-6ab67185-8eff-3a23-32fd-75bfbe37b488
Capacity:
Memory:
Value: 80Gi
Name: gpu-2
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 3
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-6b77fb80-2d68-809d-4bf1-285e5f47dcc5
Capacity:
Memory:
Value: 80Gi
Name: gpu-3
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 8
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-091ffa42-7640-9cfd-c78e-4fbb544c2a00
Capacity:
Memory:
Value: 80Gi
Name: gpu-8
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 0
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-4cbf87f3-433e-6717-5588-c33e6886832f
Capacity:
Memory:
Value: 80Gi
Name: gpu-0
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 1
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-58bd415e-dee8-f0a5-ca03-02d000554b1a
Capacity:
Memory:
Value: 80Gi
Name: gpu-1
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 4
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-417d66cd-4546-0786-59a3-ef7eb54c564d
Capacity:
Memory:
Value: 80Gi
Name: gpu-4
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 5
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-f0fdf728-dccb-f484-bbf5-33f63a90b820
Capacity:
Memory:
Value: 80Gi
Name: gpu-5
Attributes:
Driver Version:
Version: 1.0.0
Index:
Int: 6
Model:
String: LATEST-GPU-MODEL
Uuid:
String: gpu-121b8219-b8d6-015c-b2eb-1e320ee07510
Capacity:
Memory:
Value: 80Gi
Name: gpu-6
Driver: gpu.example.com
Node Name: kind-worker
Pool:
Generation: 1
Name: kind-worker
Resource Slice Count: 1
Events: <none>Create a ResourceClaim to claim the DeviceClass. First look at the ResourceClaim manifest:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/resourceclaim.yamlOutput:
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
name: example-resource-claim
namespace: dra-tutorial
spec:
devices:
requests:
- name: example-gpu
exactly:
deviceClassName: gpu.example.com
allocationMode: ExactCount
count: 1
selectors:
- cel:
expression: |-
device.capacity["gpu.example.com"].memory == quantity("80Gi")This manifest creates ResourceClaim that requests for 1 device in the gpu.example.com DeviceClass that have 80Gi of capacity.
alloctionMode defines how devices are allocated, the options are ExactCount or All in which requests for all matching devices in the pool will be allocated.
Create the ResourceClaim:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/resourceclaim.yamlOutput:
resourceclaim.resource.k8s.io/example-resource-claim createdLet's check the state of the ResourceClaim:
kubectl get resourceclaim -n dra-tutorialOutput:
NAME STATE AGE
example-resource-claim pending 28sDescribe the ResourceClaim:
kubectl describe resourceclaim -n dra-tutorial example-resource-claimOutput:
Name: example-resource-claim
Namespace: dra-tutorial
Labels: <none>
Annotations: <none>
API Version: resource.k8s.io/v1
Kind: ResourceClaim
Metadata:
Creation Timestamp: 2025-11-10T18:38:11Z
Resource Version: 16990
UID: a0db2f1d-f8e8-4143-a5c2-d29877cc2d8a
Spec:
Devices:
Requests:
Exactly:
Allocation Mode: ExactCount
Count: 1
Device Class Name: gpu.example.com
Selectors:
Cel:
Expression: device.capacity["gpu.example.com"].memory == quantity("80Gi")
Name: example-gpu
Status:
Events: <none>Ollama (Omni-Layer Learning Language Acquisition Model) is a tool that runs LLMs locally.
Ollama.com hosts many LLM models.
Let's deploy an Ollama Pod with the alpine/ollama container image which is a minimal CPU-only image.
If you're using a GPU, you can use the ollama/ollama container image to take advantage of the GPU.
Let's take a look at the Pod manifest:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/pod.yamlOutput:
apiVersion: v1
kind: Pod
metadata:
name: ollama
namespace: dra-tutorial
labels:
app: ollama
spec:
containers:
- name: ollama
image: docker.io/alpine/ollama:0.12.9
ports:
- name: http
containerPort: 11434
protocol: TCP
resources:
claims:
- name: gpu
resourceClaims:
- name: gpu
resourceClaimName: example-resource-claimCreate the Pod:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/pod.yamlOutput
pod/ollama createdCheck the state of the ResourceClaim:
kubectl get resourceclaim -n dra-tutorialOutput:
NAME STATE AGE
example-resource-claim allocated,reserved 5mNow the ResourceClaim is at an allocated,reserved.
Describe the ResourceClaim:
kubectl describe resourceclaim -n dra-tutorial example-resource-claimOutput:
Name: example-resource-claim
Namespace: dra-tutorial
Labels: <none>
Annotations: <none>
API Version: resource.k8s.io/v1
Kind: ResourceClaim
Metadata:
Creation Timestamp: 2025-11-10T18:38:11Z
Finalizers:
resource.kubernetes.io/delete-protection
Resource Version: 18455
UID: a0db2f1d-f8e8-4143-a5c2-d29877cc2d8a
Spec:
Devices:
Requests:
Exactly:
Allocation Mode: ExactCount
Count: 1
Device Class Name: gpu.example.com
Selectors:
Cel:
Expression: device.capacity["gpu.example.com"].memory == quantity("80Gi")
Name: example-gpu
Status:
Allocation:
Devices:
Results:
Device: gpu-7
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Node Selector:
Node Selector Terms:
Match Fields:
Key: metadata.name
Operator: In
Values:
kind-worker
Reserved For:
Name: ollama
Resource: pods
UID: bc92ce32-b3fa-4632-bbe5-54c4553dbd1d
Events: <none>The ResourceClaim shows that the ollama Pod reserved gpu-7.
It reserves only one GPU since the ResourceClaim had allocationMode: ExactCount and count: 1.
If the ResourceClaim had allocationMode: All, then the ResourceClaim would claim all devices that satisfies the request selection CEL expresion and the output would be similar to the following in which the ResouceClaim claims 9 GPUs for the Pod:
Name: example-resource-claim
Namespace: dra-tutorial
Labels: <none>
Annotations: <none>
API Version: resource.k8s.io/v1
Kind: ResourceClaim
Metadata:
Creation Timestamp: 2025-11-06T07:54:29Z
Finalizers:
resource.kubernetes.io/delete-protection
Resource Version: 7552
UID: 45f7a77f-07c4-465c-9f62-c90b9e3b4ba5
Spec:
Devices:
Requests:
Exactly:
Allocation Mode: All
Device Class Name: gpu.example.com
Selectors:
Cel:
Expression: device.capacity["gpu.example.com"].memory == quantity("80Gi")
Name: example-gpu
Status:
Allocation:
Devices:
Results:
Device: gpu-2
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Device: gpu-3
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Device: gpu-4
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Device: gpu-5
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Device: gpu-6
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Device: gpu-0
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Device: gpu-1
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Device: gpu-7
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Device: gpu-8
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Node Selector:
Node Selector Terms:
Match Fields:
Key: metadata.name
Operator: In
Values:
kind-worker
Reserved For:
Name: ollama
Resource: pods
UID: 69dc4953-e0ad-4955-afda-c5a8fc6b7005
Events: <none>Any additional Pods that use this ResourceClaim are also listed under Reserved For:
Check how the DRA driver handled the device allocation:
kubectl logs -l app.kubernetes.io/name=dra-example-driver -n dra-tutorialOutput:
I1110 18:31:44.155296 1 health.go:70] "connecting to registration socket" path="unix:///var/lib/kubelet/plugins_registry/gpu.example.com-reg.sock"
I1110 18:31:44.155616 1 health.go:83] "connecting to DRA socket" path="unix:///var/lib/kubelet/plugins/gpu.example.com/dra.sock"
I1110 18:31:44.156535 1 health.go:103] "starting healthcheck service" addr="[::]:51515"
I1110 18:54:37.680486 1 driver.go:107] PrepareResourceClaims is called: number of claims: 1
I1110 18:54:37.683431 1 driver.go:134] Returning newly prepared devices for claim 'a0db2f1d-f8e8-4143-a5c2-d29877cc2d8a': [{[example-gpu] kind-worker gpu-7 [k8s.gpu.example.com/gpu=common k8s.gpu.example.com/gpu=a0db2f1d-f8e8-4143-a5c2-d29877cc2d8a-gpu-7]}]Check the Pod:
kubectl get pods -n dra-tutorial -l app=ollamaOutput:
NAME READY STATUS RESTARTS AGE
ollama 1/1 Running 0 2m5sCheck the environment variables in the Ollama Pod:
kubectl -n dra-tutorial exec ollama -- envOutput:
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
HOSTNAME=ollama
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
OLLAMA_HOST=0.0.0.0
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_NODE_NAME=kind-worker
DRA_RESOURCE_DRIVER_NAME=gpu.example.com
GPU_DEVICE_7_RESOURCE_CLAIM=a0db2f1d-f8e8-4143-a5c2-d29877cc2d8a
GPU_DEVICE_7=gpu-7
GPU_DEVICE_7_SHARING_STRATEGY=TimeSlicing
GPU_DEVICE_7_TIMESLICE_INTERVAL=Default
HOME=/rootEnvironment variables are set in each container to indicate their accessible GPU(s) and how the GPUs would be configured.
Time slicing allows multiple workloads to share a GPU by alternating execution time
In the Ollama Pod, pull Meta's llama 3.2 LLM:
kubectl -n dra-tutorial exec ollama -- ollama pull llama3.2Output:
pulling manifest
pulling dde5aa3fc5ff: 100% ▕██████████████████▏ 2.0 GB
pulling 966de95ca8a6: 100% ▕██████████████████▏ 1.4 KB
pulling fcc5a6bec9da: 100% ▕██████████████████▏ 7.7 KB
pulling a70ff7e570d9: 100% ▕██████████████████▏ 6.0 KB
pulling 56bb8bd477a5: 100% ▕██████████████████▏ 96 B
pulling 34bb5ab01051: 100% ▕██████████████████▏ 561 B
verifying sha256 digest
writing manifest
success Test the Llama model (this will take about 4 minutes):
kubectl -n dra-tutorial exec ollama -- ollama run llama3.2 "what is kubernetes"Output (output may vary):
Kubernetes (also known as K8s) is an open-source container orchestration system for automating the deployment, scaling, and management of containerized applications. It was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation (CNCF).
Kubernetes provides a platform-agnostic way to deploy, manage, and scale applications in a cloud-native environment, which allows for greater flexibility, scalability, and reliability. Here are some key features of Kubernetes:
**Key Features:**
1. **Container Orchestration:** Kubernetes manages the lifecycle of containers across multiple hosts, ensuring that containers are deployed, scaled, and terminated as needed.
2. **Self-healing:** If a container fails or is terminated, Kubernetes automatically redeployes it to ensure high availability.
3. **Scaling:** Kubernetes allows you to scale your applications horizontally (add more replicas) or vertically (increase resources).
4. **Resource Management:** Kubernetes manages resource allocation and deallocation, ensuring that resources are used efficiently.
5. **Networking:** Kubernetes provides a built-in networking system for containers, allowing them to communicate with each other.
6. **Service Discovery:** Kubernetes provides a service discovery mechanism, making it easy to find and access services.
7. **Security:** Kubernetes provides a secure environment by isolating applications and limiting access.
**Components:**
1. **Pods:** The basic execution unit in Kubernetes, consisting of one or more containers.
2. **ReplicaSets:** Ensures that a specified number of replicas (i.e., copies) of a Pod are running at any given time.
3. **Deployments:** Manages the rollout of new versions of an application.
4. **Services:** Provides a stable network identity for accessing applications.
5. **Persistent Volumes:** Provides persistent storage for data that needs to be retained across pod restarts.
**Benefits:**
1. **Increased Efficiency:** Automates many tasks, freeing up time and resources.
2. **Improved Scalability:** Easily scale applications horizontally or vertically.
3. **High Availability:** Ensures high availability by deploying multiple replicas of an application.
4. **Simplified Management:** Provides a single platform for managing multiple applications.
**Use Cases:**
1. **Web Applications:** Kubernetes is well-suited for web applications with complex dependencies and traffic patterns.
2. **Microservices Architecture:** Kubernetes is ideal for microservices architecture, where each service is self-contained and needs to be deployed independently.
3. **Cloud-Native Applications:** Kubernetes provides a platform-agnostic way to deploy cloud-native applications.
In summary, Kubernetes is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. It provides a flexible, scalable, and secure environment for building cloud-native applications.Meta's Llama 3.2 (3B parameters) collection of LLMs is 2.0 GB.
Let's use a smaller model like tinyllama.
In the Ollama Pod, pull the tinyllama LLM:
kubectl -n dra-tutorial exec ollama -- ollama pull tinyllama:1.1bOutput:
pulling manifest
pulling 2af3b81862c6: 100% ▕██████████████████▏ 637 MB
pulling af0ddbdaaa26: 100% ▕██████████████████▏ 70 B
pulling c8472cd9daed: 100% ▕██████████████████▏ 31 B
pulling fa956ab37b8c: 100% ▕██████████████████▏ 98 B
pulling 6331358be52a: 100% ▕██████████████████▏ 483 B
verifying sha256 digest
writing manifest
success Run the same query:
kubectl -n dra-tutorial exec ollama -- ollama run tinyllama:1.1b "what is kubernetes"Output:
Kubernetes is a popular open-source container orchestration system that allows you to manage and deploy containers in a highly available and scalable way. It is designed for automating the deployment, management, and scaling of containerized applications at scale in large-scale enterprise environments. Kubernetes offers several benefits over traditional container management approaches such as:
1. Flexibility: Kubernetes allows you to define your application requirements, and it manages container resources dynamically based on those requirements. This eliminates the need for manual container resource allocation and scaling.
2. Scalability: Kubernetes enables you to automatically scale up or down your applications' containers based on demand. This feature is called Autoscaling and is part of its core functionality.
3. Separation of Concerns: Kubernetes separates the deployment, management, and scaling aspects of containerized applications from their underlying infrastructure components. This helps you to maintain a more reliable and scalable system without the need for additional tools or services.
4. Scalability: You can scale your containers up or down by simply updating the number of replicas for the Pods associated with each container within Kubernetes, which is an on-demand provisioning model. This provides you with flexibility and control over how many containers are running at any given time.
5. Integration: Kubernetes provides a plethora of integrations with other Open Source technologies like Docker Swarm or ECS in AWS, making it easy to integrate with existing infrastructure as well.
Overall, Kubernetes is a powerful container orchestration platform that offers several benefits and features that can help you manage your containers more effectively and efficiently.How do the responses differ?
Let's add another Pod that references the same ResourceClaim.
Let's look at the Pod manifest:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/pod2.yamlOutput:
apiVersion: v1
kind: Pod
metadata:
name: ollama2
namespace: dra-tutorial
labels:
app: ollama2
spec:
containers:
- name: ollama2
image: alpine/ollama:0.11.10
ports:
- name: http
containerPort: 11434
protocol: TCP
resources:
claims:
- name: gpu
resourceClaims:
- name: gpu
resourceClaimName: example-resource-claimCreate the Pod:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/pod2.yamlOutput
pod/ollama2 createdDescribe the ResourceClaim:
kubectl describe resourceclaim -n dra-tutorial example-resource-claimOutput:
Name: example-resource-claim
Namespace: dra-tutorial
Labels: <none>
Annotations: <none>
API Version: resource.k8s.io/v1
Kind: ResourceClaim
Metadata:
Creation Timestamp: 2025-11-10T18:38:11Z
Finalizers:
resource.kubernetes.io/delete-protection
Resource Version: 26739
UID: a0db2f1d-f8e8-4143-a5c2-d29877cc2d8a
Spec:
Devices:
Requests:
Exactly:
Allocation Mode: ExactCount
Count: 1
Device Class Name: gpu.example.com
Selectors:
Cel:
Expression: device.capacity["gpu.example.com"].memory == quantity("80Gi")
Name: example-gpu
Status:
Allocation:
Devices:
Results:
Device: gpu-7
Driver: gpu.example.com
Pool: kind-worker
Request: example-gpu
Node Selector:
Node Selector Terms:
Match Fields:
Key: metadata.name
Operator: In
Values:
kind-worker
Reserved For:
Name: ollama
Resource: pods
UID: bc92ce32-b3fa-4632-bbe5-54c4553dbd1d
Name: ollama2
Resource: pods
UID: aa09d590-c5b1-4c1e-87c8-730279af6bd2
Events: <none>Check the environment variables for the new Pod:
kubectl -n dra-tutorial exec ollama2 -- envOutput:
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
HOSTNAME=ollama2
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
OLLAMA_HOST=0.0.0.0
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_NODE_NAME=kind-worker
DRA_RESOURCE_DRIVER_NAME=gpu.example.com
GPU_DEVICE_7_RESOURCE_CLAIM=a0db2f1d-f8e8-4143-a5c2-d29877cc2d8a
GPU_DEVICE_7=gpu-7
GPU_DEVICE_7_SHARING_STRATEGY=TimeSlicing
GPU_DEVICE_7_TIMESLICE_INTERVAL=Default
HOME=/rootIn this case, both Pods are sharing GPU-7
We can use a ResourceClaimTemplate to dynamically create ResourceClaims per-Pod.
In this case, the ResourceClaims are not shareable.
Let's look at the ResourceClaimTemplate manifest:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/resourceclaimtemplate.yamlOutput:
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: example-resource-claim-template
namespace: dra-tutorial
spec:
spec:
devices:
requests:
- name: req-0
firstAvailable:
- name: 80gi-gpu
deviceClassName: gpu.example.com
allocationMode: ExactCount
count: 1
selectors:
- cel:
expression: |-
device.capacity["gpu.example.com"].memory == quantity("80Gi")This ResourceClaimTemplate requests for one device with 80Gi memory.
Create the ResourceClaimTemplate:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/resourceclaimtemplate.yamlOutput:
resourceclaimtemplate.resource.k8s.io/example-resource-claim-template createdTake a look into the ResourceClaimTemplate with kubectl describe:
kubectl describe resourceclaimtemplate -n dra-tutorialOutput:
Name: example-resource-claim-template
Namespace: dra-tutorial
Labels: <none>
Annotations: <none>
API Version: resource.k8s.io/v1
Kind: ResourceClaimTemplate
Metadata:
Creation Timestamp: 2025-11-10T23:05:21Z
Resource Version: 40903
UID: 51a99a22-fdcd-43db-a299-d9af28b9eaf0
Spec:
Metadata:
Spec:
Devices:
Requests:
First Available:
Allocation Mode: ExactCount
Count: 1
Device Class Name: gpu.example.com
Name: 80gi-gpu
Selectors:
Cel:
Expression: device.capacity["gpu.example.com"].memory == quantity("80Gi")
Name: req-0
Events: <none>Create multiple Pods that reference the ResourceClaimTemplate.
Use a Job to create multiple Pods that refers to the same ResourceClaimTemplate.
Take a look at the Job manifest:
curl -w "\n" https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/job.yamlOutput:
apiVersion: batch/v1
kind: Job
metadata:
name: dra-job
namespace: dra-tutorial
spec:
completions: 4
parallelism: 2
template:
spec:
restartPolicy: Never
containers:
- name: busybox
image: busybox:1.37.0
command: ["/bin/sh","-c","env && sleep 600"]
resources:
claims:
- name: my-gpu-claim
resourceClaims:
- name: my-gpu-claim
resourceClaimTemplateName: example-resource-claim-templateCreate the Job:
kubectl apply -f https://raw.githubusercontent.com/cloudnativeessentials/dra-tutorial/refs/heads/main/manifests/job.yamlOutput:
job.batch/dra-job createdkubectl -n dra-tutorial get jobs,pods,resourceclaimsOutput:
NAME STATUS COMPLETIONS DURATION AGE
job.batch/dra-job Running 0/4 90s 90s
NAME READY STATUS RESTARTS AGE
pod/dra-example-driver-kubeletplugin-nzz5z 1/1 Running 0 27m
pod/dra-job-64mw7 1/1 Running 0 90s
pod/dra-job-8vh8x 1/1 Running 0 90s
pod/ollama 1/1 Running 0 27m
pod/ollama2 1/1 Running 0 12s
NAME STATE AGE
resourceclaim.resource.k8s.io/dra-job-64mw7-my-gpu-claim-9p7df allocated,reserved 90s
resourceclaim.resource.k8s.io/dra-job-8vh8x-my-gpu-claim-ttmhm allocated,reserved 90s
resourceclaim.resource.k8s.io/example-resource-claim allocated,reserved 27mDescribe each of the new ResourceClaims:
kubectl describe $(kubectl get resourceclaim -o name -n dra-tutorial | grep "dra-job") -n dra-tutorialOutput:
Name: dra-job-64mw7-my-gpu-claim-9p7df
Namespace: dra-tutorial
Labels: <none>
Annotations: resource.kubernetes.io/pod-claim-name: my-gpu-claim
API Version: resource.k8s.io/v1
Kind: ResourceClaim
Metadata:
Creation Timestamp: 2025-11-11T00:58:46Z
Finalizers:
resource.kubernetes.io/delete-protection
Generate Name: dra-job-64mw7-my-gpu-claim-
Owner References:
API Version: v1
Block Owner Deletion: true
Controller: true
Kind: Pod
Name: dra-job-64mw7
UID: 592011d0-4b37-48f4-be02-fb9d64d0ad81
Resource Version: 27796
UID: abe20b78-e6ca-4cec-8828-753fe16f1226
Spec:
Devices:
Requests:
First Available:
Allocation Mode: ExactCount
Count: 1
Device Class Name: gpu.example.com
Name: 80gi-gpu
Selectors:
Cel:
Expression: device.capacity["gpu.example.com"].memory == quantity("80Gi")
Name: req-0
Status:
Allocation:
Devices:
Results:
Device: gpu-1
Driver: gpu.example.com
Pool: kind-worker
Request: req-0/80gi-gpu
Node Selector:
Node Selector Terms:
Match Fields:
Key: metadata.name
Operator: In
Values:
kind-worker
Reserved For:
Name: dra-job-64mw7
Resource: pods
UID: 592011d0-4b37-48f4-be02-fb9d64d0ad81
Events: <none>
Name: dra-job-8vh8x-my-gpu-claim-ttmhm
Namespace: dra-tutorial
Labels: <none>
Annotations: resource.kubernetes.io/pod-claim-name: my-gpu-claim
API Version: resource.k8s.io/v1
Kind: ResourceClaim
Metadata:
Creation Timestamp: 2025-11-11T00:58:46Z
Finalizers:
resource.kubernetes.io/delete-protection
Generate Name: dra-job-8vh8x-my-gpu-claim-
Owner References:
API Version: v1
Block Owner Deletion: true
Controller: true
Kind: Pod
Name: dra-job-8vh8x
UID: 0f914d86-8360-4efc-ada3-f0fc81dab095
Resource Version: 27804
UID: 9d83255c-724d-40ba-9412-1cc6e6b28f88
Spec:
Devices:
Requests:
First Available:
Allocation Mode: ExactCount
Count: 1
Device Class Name: gpu.example.com
Name: 80gi-gpu
Selectors:
Cel:
Expression: device.capacity["gpu.example.com"].memory == quantity("80Gi")
Name: req-0
Status:
Allocation:
Devices:
Results:
Device: gpu-2
Driver: gpu.example.com
Pool: kind-worker
Request: req-0/80gi-gpu
Node Selector:
Node Selector Terms:
Match Fields:
Key: metadata.name
Operator: In
Values:
kind-worker
Reserved For:
Name: dra-job-8vh8x
Resource: pods
UID: 0f914d86-8360-4efc-ada3-f0fc81dab095
Events: <none>In this example, the ResourceClaims from the ResourceClaimTemplate used different GPUs: gpu-1 and gpu-2 (your results will vary).
Confirm with the environment variables from the Job Pods:
kubectl get pods -l job-name=dra-job -n dra-tutorial -o nameOutput:
pod/dra-job-64mw7
pod/dra-job-8vh8xExec into each Job Pod:
kubectl -n dra-tutorial exec pod/dra-job-64mw7 -- envOutput:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=dra-job-64mw7
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_NODE_NAME=kind-worker
DRA_RESOURCE_DRIVER_NAME=gpu.example.com
GPU_DEVICE_1_RESOURCE_CLAIM=abe20b78-e6ca-4cec-8828-753fe16f1226
GPU_DEVICE_1=gpu-1
GPU_DEVICE_1_SHARING_STRATEGY=TimeSlicing
GPU_DEVICE_1_TIMESLICE_INTERVAL=Default
HOME=/rootExec into the other Job Pod:
kubectl -n dra-tutorial exec pod/dra-job-8vh8x -- envOutput:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=dra-job-8vh8x
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_NODE_NAME=kind-worker
DRA_RESOURCE_DRIVER_NAME=gpu.example.com
GPU_DEVICE_2_RESOURCE_CLAIM=9d83255c-724d-40ba-9412-1cc6e6b28f88
GPU_DEVICE_2=gpu-2
GPU_DEVICE_2_SHARING_STRATEGY=TimeSlicing
GPU_DEVICE_2_TIMESLICE_INTERVAL=Default
HOME=/rootWe can see that one Job Pod is allocated GPU-1 and the second Job Pod is allocated GPU-2 in this example.
The Jobs will complete to run until 4 Job Pods complete.
Not all CPUs are created equal, some have higher performance, some have better cache.
With a DRA driver, we can use a DRA driver to scans the Node through the Node Resource Interface and publish a ResourceSlice with each CPU represented as a device along with attributes like core ID, coreType, socktID, numaNode.
DRA Drive for CPU resources can be used to increase performance or have precise CPU requests:
- align resources (CPU, GPU, NIC) on the same NUMA domain
- request CPUs that are high performance cores or all cores sharing the same L3 cache
Learn more on the GitHub repo for DRA Driver for CPU Resources.
For AMD Instinct GPUs and Radeon GPUs, AMD's DRA driver is at https://github.com/ROCm/k8s-gpu-dra-driver
For FuriosaAI NPU which is used for deep learning inferencing, the DRA driver is at https://github.com/furiosa-ai/furiosa-dra-driver-guide