HAMi’s CNCF Incubation Shows Why Kubernetes Still Needs GPU-Sharing Middleware

The Cloud Native Computing Foundation’s Technical Oversight Committee has accepted HAMi, an open source GPU virtualization project for Kubernetes, as a CNCF incubating project. HAMi’s own CNCF project page dates the move to July 2, 2026; the foundation’s blog post publicizing it went up July 15.

HAMi (Heterogeneous AI Computing Virtualization Middleware, formerly known as k8s-vGPU-scheduler) solves a problem Kubernetes still does not handle well by itself: letting more than one workload share an expensive GPU without wasting most of its capacity. Its incubation lands at a specific moment, just as Kubernetes’ own native answer to fine-grained device sharing, Dynamic Resource Allocation (DRA), has reached general availability. That timing raises an obvious question for platform teams: if Kubernetes itself can now describe GPUs in detail, why does a third-party middleware project still need its own governance track and roadmap?

The Problem: Kubernetes Still Treats a GPU as All or Nothing

According to HAMi’s own engineering documentation, a typical AI inference service uses only 20 to 40 percent of a GPU’s compute capacity and a small slice of its memory, leaving the rest idle. On cheap hardware that would be a minor inefficiency. On data center GPUs costing tens of thousands of dollars each, it is a direct cost problem.

The root cause, per the same documentation, is how Kubernetes’ Device Plugin extension mechanism works. A Device Plugin runs as a DaemonSet on each GPU node, registers with the kubelet, and reports available devices to the API server as plain integers (for example, nvidia.com/gpu: 1). That interface has no concept of a VRAM quota or a fractional core count. A pod either gets an entire physical GPU or it waits for one to free up, regardless of how little of that GPU its workload actually needs.

Time-Slicing and MIG Cover Part of the Gap, Not All of It

Two existing approaches soften this limitation, and HAMi’s documentation is candid about where each one falls short. NVIDIA’s own Time-Slicing lets multiple pods run concurrently on one GPU, but it enforces no memory isolation between them: if one pod’s process runs out of memory, it can crash every other workload sharing that card. NVIDIA’s MIG (Multi-Instance GPU) hardware partitioning provides real isolation at the silicon level, but only on datacenter-class cards such as the A100 and H100, which rules it out for older or lower-tier hardware and for accelerators from other vendors entirely.

HAMi’s pitch is a middle path: software-level isolation that works without new hardware and without locking a cluster to a single vendor’s top-tier cards.

How HAMi Slices a GPU Without Touching the Driver

Rather than waiting on hardware vendors or changing application code, HAMi intercepts calls at the CUDA API layer. Per the CNCF blog’s description of the project’s components, HAMi is built from a Mutating Webhook that intercepts pod submissions in the Kubernetes API server and rewrites scheduler fields and resource requests; a Scheduler Extender that filters, scores, and binds pods to nodes and devices using binpack, spread, and topology-aware placement policies; vendor-specific Device Plugins that register accelerators with Kubernetes and allocate fractional device resources to containers; and HAMi-Core, the in-container virtualization layer that enforces hard runtime limits on GPU memory and compute by intercepting the native CUDA driver on NVIDIA devices.

HAMi’s documentation describes that interception in more detail: a dynamic library, libvgpu.so, gets mounted into a container during the Device Plugin’s allocation step and overrides dlsym to hijack the NVIDIA library calls an application makes, enforcing whatever memory and compute limits were assigned to that pod. A workload that tries to allocate more VRAM than it was granted gets an out-of-memory error instead of quietly taking capacity from its neighbors. Operators declare what a pod needs using two custom resource names, nvidia.com/gpumem for VRAM in mebibytes and nvidia.com/gpucores for a compute percentage share, rather than the single whole-device integer Kubernetes understands natively.

Beyond the scheduling internals, CNCF’s write-up also credits the project with HAMi-WebUI, a visual interface for cluster and device management that gives operators visibility into allocation and utilization across a fleet, and an observability layer built on a Prometheus-compatible metrics endpoint with sample Grafana dashboards for tracking accelerator usage cluster-wide. The full source is maintained in the project’s GitHub repository.

What CNCF’s Incubation Vote Actually Confirms

CNCF grades projects on three maturity levels. By the foundation’s own definitions, Sandbox covers experimental projects not yet widely tested in production; Incubating covers projects already used successfully in production by a small number of users with a healthy pool of contributors; Graduated is reserved for projects considered stable, widely adopted, and production ready, attracting thousands of contributors. Moving up a level requires a Technical Oversight Committee vote, not just a maintainer’s say-so.

HAMi entered CNCF Sandbox on August 21, 2024. The metrics behind its move to Incubating show a project well past its early stage, though the exact figures depend on which CNCF source is measuring and when. The CNCF blog post accompanying the announcement puts the project at 2,687 contributors, up 43 percent year over year at the time it was written, and more than 550 contributing organizations. CNCF’s live project page, powered by its LFX Insights dashboard, now shows 2,851 contributors (up 48 percent year over year) and 587 contributing organizations (up 56 percent), current as of this writing. Both figures come directly from CNCF; the gap between them simply reflects a fast-growing project measured a few weeks apart. That same live dashboard rates HAMi’s automated health score at 83 out of 100, in its “Excellent” band. The codebase traces to a first commit on March 15, 2020, and the current stable release is v2.9.0.

CNCF’s announcement also points to concrete production use behind those numbers: five independent CNCF case studies have been published on HAMi so far, including DaoCloud’s deployment of the project across more than 10,000 GPUs in over 10 data centers in mainland China and Hong Kong, and China Merchants Bank’s use of HAMi to manage diverse accelerator resources at scale. “Seeing HAMi reach Incubation is a proud moment for all of us,” HAMi maintainer Xiao Zhang said in CNCF’s announcement. “HAMi now supports dozens of Heterogeneous GPU and has grown into a global community with hundreds of contributors and hundreds of end users. What makes this milestone meaningful is not just the technology, but the real ecosystem momentum behind it.”

The DRA Question: Does This Middleware Have an Expiration Date?

Kubernetes’ own long-term answer to heterogeneous device scheduling is Dynamic Resource Allocation. DRA introduces a set of APIs, ResourceClaim, DeviceClass, and ResourceSlice, that let a device driver describe its hardware’s real attributes directly to the scheduler and let users filter for things like a minimum VRAM size or a specific GPU model using CEL expressions. Per HAMi’s own documentation, DRA was promoted to general availability in Kubernetes v1.34.

HAMi’s own team is not ignoring that shift. Its documentation states plainly that HAMi still runs on the older Device Plugin model today, and that the team has already launched a separate HAMi-DRA subproject (currently v0.1.0, requiring Kubernetes 1.34 or newer) that converts HAMi’s GPU resource requests into native DRA ResourceClaim objects through a Mutating Webhook. By the project’s own description, that subproject exists as a transitional bridge toward DRA, not a permanent second architecture. CNCF’s roadmap summary for the project reinforces the same point from a different angle: alongside plans for gang-scheduling, preemption, and autoscaling, and expanding device support to AMD’s MI series and PPU accelerators, HAMi’s maintainers list “providing a solution for monitoring DRA consumption” as a committed near-term priority, alongside closer collaboration with other CNCF-adjacent scheduling projects such as KAI-scheduler, Koordinator, Kueue, llm-d, and Volcano.

That is arguably the more interesting story than the incubation vote itself. A CNCF Incubating project, backed by hundreds of contributing organizations, is treating Kubernetes’ native scheduling API as its eventual destination rather than its rival. For platform teams running shared GPU inference workloads today, that is a reason to adopt HAMi’s current Device Plugin based approach now, while watching the HAMi-DRA subproject and their own cluster’s path to Kubernetes 1.34 and newer as the migration route once DRA support matures across GPU vendors’ drivers, rather than treating either option as a permanent architectural bet.