Run Slurm on Kubernetes With Slinky: The GPU Cluster Checks That Matter
A practical Learning Hub guide to understanding Slinky on Kubernetes: split node pools, shared NFS, Multus RDMA, custom Slurm images, and the NCCL validation that proves the fabric is really working.
Slinky is useful because it does not ask GPU teams to choose between Kubernetes operations and Slurm job scheduling. It lets Kubernetes run the lifecycle of the control-plane pods while Slurm still gives researchers and training jobs the sbatch, srun, partitions, and node semantics they already know.
Table Of Content
- What Slinky actually gives you
- Use split node pools
- Shared storage is part of the cluster design
- Do not skip NFS performance tuning
- RDMA requires more than a GPU node
- NetworkAttachmentDefinitions are not decoration
- Validate what transport NCCL used
- Custom images are a reliability choice
- Tighten convenience before production
- Readiness checklist
- Production acceptance tests before onboarding users
- Prerequisites to verify explicitly
- Target systems: Kubernetes control plane and Slurm login pod
- Storage and fabric validation
- Failure drills that catch hidden coupling
The DigitalOcean tutorial walks through Slinky on DOKS with GPU nodes, managed NFS, Multus-attached RDMA interfaces, and a multi-node NCCL validation. This guide translates that into the operational checks you should understand before treating a Slinky cluster as ready for real training work.
What Slinky actually gives you
Slinky is SchedMD’s Kubernetes operator for Slurm. It runs components such as slurmctld, slurmd, login pods, and optional accounting pieces as Kubernetes resources. Slurm remains the workload manager; Kubernetes provides pod lifecycle, placement, restarts, and integration with cluster services.
Use split node pools
A practical design separates small CPU management nodes from expensive GPU workers. The operator, webhooks, cert-manager, and supporting control-plane pods should not compete with training jobs for GPU capacity. GPU nodes should carry the vendor taints and labels that keep ordinary workloads away.
Shared storage is part of the cluster design
HPC-style workflows usually assume a login node writes job scripts and outputs to shared storage while workers read and write the same paths. On DOKS, the tutorial uses managed NFS with ReadWriteMany semantics so login and worker pods see the same workspace.
Do not skip NFS performance tuning
DigitalOcean’s NFS guidance warns that GPU nodes need network tuning before NFS mounts are created. If a pod mounts NFS while the interface is still at the default MTU, throughput can stay degraded for the life of that mount. The safer pattern is a DaemonSet that tunes the node, labels it ready, and lets NFS workloads schedule only after that label exists.
RDMA requires more than a GPU node
Distributed training needs fast GPU-to-GPU communication across nodes. Kubernetes pods normally receive one default network interface, so specialized fabric NICs require extra plumbing. Multus CNI provides the multi-network attachment model that lets selected pods receive additional interfaces.
NetworkAttachmentDefinitions are not decoration
Each fabric interface needs a matching NetworkAttachmentDefinition in the namespace where the Slurm worker pods run. If those resources are missing, in the wrong namespace, or not requested by the chart values, the job may still schedule while NCCL quietly falls back to slower transport.
Validate what transport NCCL used
NVIDIA’s NCCL documentation describes collective communication such as all-reduce and the role of high-bandwidth interconnects. For a Slinky GPU cluster, a multi-node NCCL all-reduce is the practical smoke test. You want debug output showing the expected fabric devices and bandwidth that matches RDMA, not single-digit GB/s TCP fallback.
Custom images are a reliability choice
The worker image needs the CUDA runtime, NCCL, MPI support, RDMA tools, and benchmark utilities that can prove the fabric works. The login image needs the developer basics: shell tools, editor, Git, Python, curl, and enough package tooling for users to prepare jobs without leaving the login pod.
Tighten convenience before production
Getting-started images often include broad permissions for speed. Before handing the cluster to real users, review sudo access, image provenance, registry permissions, user mapping, and whether jobs can write only to the intended shared paths.
Readiness checklist
- Management pods run on CPU nodes; training workloads land only on GPU nodes.
- Shared NFS is mounted read/write by login and worker pods.
- GPU nodes are network-tuned before workloads mount NFS.
- Multus and fabric NetworkAttachmentDefinitions exist in the Slurm namespace.
- Worker images include CUDA, NCCL, MPI, and RDMA diagnostic tools.
- A multi-node NCCL all-reduce proves fabric transport and expected bandwidth.
- Monitoring and accounting are planned before multi-user production use.
Slinky makes Slurm-on-Kubernetes approachable, but the cluster is only real when storage, fabric, images, and validation all line up. The proof is not that pods start. The proof is that a distributed GPU job can move data across nodes at the speed the hardware promised.
Production acceptance tests before onboarding users
A Slinky cluster should not be considered ready just because Helm finished successfully or Slurm commands return from a login pod. The production bar is higher: Kubernetes must see the GPUs, Slurm must model the nodes correctly, shared storage must survive real job I/O, and the RDMA path must be proven by a distributed workload. Slinky provides the Slurm-on-Kubernetes control model, but the readiness evidence should come from both Kubernetes and Slurm. Slinky Slurm operator Slurm quick-start administration guide
Prerequisites to verify explicitly
- GPU nodes run a compatible driver stack and expose GPU capacity to Kubernetes. Kubernetes device plugins advertise specialized hardware resources to the kubelet, and NVIDIA’s device plugin is the common path for
nvidia.com/gpuresources. Kubernetes device plugins NVIDIA Kubernetes device plugin - GPU node taints and tolerations keep non-training pods away from expensive workers. Kubernetes taints and tolerations
- The Slurm namespace contains the Multus NetworkAttachmentDefinitions required for each fabric interface. Multus is the common Kubernetes multi-network attachment mechanism used when pods need more than the default interface. Multus CNI
- Shared storage is mounted with the same UID, GID, and path expectations from login and worker pods.
slurm.confdescribes partitions, nodes, CPUs, memory, and GPU generic resources consistently with the Kubernetes node pool. Slurm slurm.conf reference
Target systems: Kubernetes control plane and Slurm login pod
kubectl get nodes -L nvidia.com/gpu.product
kubectl describe node <gpu-node> | grep -E 'nvidia.com/gpu|Taints'
kubectl -n slurm get network-attachment-definitions
kubectl -n slurm get pods -o wide
sinfo -N -o "%N %t %G %m"
scontrol show node
srun -N2 hostname
srun -N1 --gres=gpu:1 nvidia-smiThe Kubernetes checks prove that hardware and networking resources are visible to pods. The Slurm checks prove that Slurm sees the same capacity and can allocate it to jobs. If Kubernetes reports eight GPUs but Slurm exposes fewer generic resources, fix the Slurm node configuration before users start debugging failed training scripts.
Storage and fabric validation
Use a small write/read job to confirm that login and worker pods share the same filesystem view. Then run a multi-node communication benchmark before running a real model. NCCL documents environment variables for selecting and debugging network transport; during the smoke test, enable debug output and confirm that the expected RDMA devices are used instead of a TCP fallback. NCCL environment variables
# inside a Slurm batch job
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_SOCKET_IFNAME=<fabric-interface>
srun -N2 --gres=gpu:8 ./build/all_reduce_perf -b 8M -e 4G -f 2 -g 8Failure drills that catch hidden coupling
- Cordon and drain one GPU node, then confirm queued jobs reschedule instead of remaining stuck in an unclear state.
- Restart the
slurmctldpod and verify that running and queued job state is recoverable. - Temporarily block the RDMA NetworkAttachmentDefinition in a test namespace and confirm the NCCL validation fails loudly.
- Fill a test NFS quota and confirm user jobs fail with actionable errors rather than corrupting outputs.
- Run
sacct,squeue, and cluster metrics review after the test so accounting and observability are ready before multi-user production.
Featured image: “The CSIRO GPU cluster at the data centre” by CSIRO, licensed under CC BY 3.0.
No Comment! Be the first one.