Skip to content

GPUs

UTHPC Kubernetes has several NVIDIA P100 and RTX PRO 6000 Blackwell GPUs inside the cluster, which are available for workloads.

Some RTX PRO 6000 Blackwell GPUs are divided into smaller GPU instances using NVIDIA Multi-Instance GPU (MIG). This allows multiple workloads to share the same physical GPU while receiving dedicated GPU resources.

Recommendation is not to "sit" on a GPU persistently, but to have a queue and worker system for spinning up workloads only when needed. This helps with controlling the cost, as billing happens for a whole GPU.

Using a full GPU

Cluster users can ask GPUs for their workloads like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cuda-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cuda-app
  template:
    metadata:
      labels:
        app: cuda-app
    spec:
      containers:
      - name: cuda-container
        image: "k8s.gcr.io/cuda-vector-add:v0.1"
        resources:
          limits: 
            nvidia.com/gpu: 1 # (1)
      tolerations: # (2)
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
  1. This requests one full GPU and ensures the workload runs on a node with an available GPU.
  2. GPU nodes are tainted to prevent non-GPU workloads from using the GPU. GPU workloads must tolerate this taint.

Using GPU partitions (MIG)

NVIDIA Multi-Instance GPU (MIG) allows a physical GPU to be divided into smaller, isolated GPU instances.

Each instance has dedicated GPU memory and compute resources. This allows multiple users to run GPU workloads on the same physical GPU without requiring an entire GPU each.

MIG is especially useful for workloads that do not require the full memory or computing capacity of a physical GPU, such as smaller machine learning models, development, inference, and testing.

Available GPU partition sizes

The RTX PRO 6000 Blackwell GPUs support the following partition sizes in our cluster:

GPU resource GPU memory Description
nvidia.com/mig-1g.24gb 24 GB Small GPU partition
nvidia.com/mig-2g.48gb 48 GB Medium GPU partition
nvidia.com/gpu Full GPU memory Entire physical GPU

GPU memory and compute capacity are both partitioned. A 24 GB MIG instance is not equivalent to a full GPU with a 24 GB memory limit.

The cluster has GPU nodes configured with different partition sizes. Kubernetes automatically selects a node with an available instance matching the requested resource.

Important: Full GPUs and MIG partitions are separate Kubernetes resources. Requesting nvidia.com/gpu: 1 does not automatically allocate a smaller GPU partition.

Example: Requesting a 24 GB GPU partition

To request one 24 GB MIG instance, use nvidia.com/mig-1g.24gb instead of nvidia.com/gpu:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cuda-mig-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cuda-mig
  template:
    metadata:
      labels:
        app: cuda-mig
    spec:
      containers:
        - name: cuda-container
          image: nvcr.io/nvidia/cuda:13.0.0-base-ubuntu22.04
          command: ["nvidia-smi"]
          resources:
            limits:
              nvidia.com/mig-1g.24gb: 1 # (1)
      tolerations: # (2)
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      restartPolicy: Always
  1. Requests one 24 GB MIG instance. Kubernetes schedules the workload only on a node with a free instance of this size.
  2. Allows the workload to run on tainted GPU nodes.

For a long-running application, replace the example command with your application's startup command.

Example: Requesting a 48 GB GPU partition

For larger workloads that require up to 48 GB GPU memory, use:

resources:
  limits:
    nvidia.com/mig-2g.48gb: 1

No additional node selection is normally required. Kubernetes schedules the workload on a node advertising the requested GPU resource.

Choosing a GPU resource

Choose the smallest GPU resource that meets your workload's requirements.

  • 24 GB MIG (nvidia.com/mig-1g.24gb): Smaller models, notebooks, development, and testing.
  • 48 GB MIG (nvidia.com/mig-2g.48gb): Medium-sized models and workloads requiring more GPU memory or compute capacity.
  • Full GPU (nvidia.com/gpu): Large models, GPU-intensive computations, or workloads requiring an entire GPU.

Not all workloads support MIG. Some multi-GPU applications and workloads requiring particular GPU features may need full GPUs.

Checking GPU availability

To see all allocatable GPU resources on cluster nodes:

kubectl get nodes \
  -o custom-columns=\
NAME:.metadata.name,\
1GPU:.status.allocatable.nvidia\\.com/gpu,\
MIG24:.status.allocatable.nvidia\\.com/mig-1g\\.24gb,\
MIG48:.status.allocatable.nvidia\\.com/mig-2g\\.48gb

If a workload requests a GPU resource that is currently unavailable, the Pod will remain in the Pending state until a suitable resource becomes available.

Important considerations

  • GPU resources are exclusive: A MIG instance is allocated to one workload at a time unless additional GPU sharing mechanisms have been configured.
  • Request the correct resource: Kubernetes does not automatically substitute a full GPU for a MIG instance, or vice versa.
  • GPU partition sizes are fixed: A running workload cannot dynamically increase its GPU partition size. Request a larger resource if necessary.
  • GPU resources are limited: Release resources when they are no longer needed.
  • Use compatible CUDA applications: MIG workloads require compatible NVIDIA drivers, CUDA software, and application frameworks.
  • Billing: GPU partitions and full GPUs may have different resource accounting. Check the applicable UTHPC billing rules.

For additional technical information, see the NVIDIA MIG documentation and NVIDIA GPU Operator MIG guide.