Headlamp’s Kubeflow Plugin Shows Why AI Platforms Still Need a Kubernetes-Native View

A new Headlamp plugin for Kubeflow arrived on July 13, giving cluster operators a dedicated, Kubernetes-native view into Kubeflow’s machine learning workloads, from Jupyter notebooks to distributed training jobs. The plugin, written by independent contributor Alok Dangre and published on the official Kubernetes Blog, reads Kubeflow’s custom resources straight from the Kubernetes API server instead of routing through Kubeflow’s own dashboard or backend services. That distinction is the point: Kubeflow’s Central Dashboard is built for data scientists submitting experiments and pipelines, not for the operators and site reliability engineers who have to work out why a notebook is stuck or a training job failed, and until now those operators have generally had to drop back to raw kubectl commands to get an answer.

The Problem: ML Dashboards Hide the Kubernetes Layer

Kubeflow assembles its machine learning stack the Kubernetes-native way: every capability, from notebook servers to hyperparameter tuning to distributed training, is exposed as a Custom Resource Definition (CRD). In theory, that design is a gift to cluster operators, since it means ML workloads can be observed and managed with the same primitives as everything else running in the cluster.

In practice, the Kubernetes blog post introducing the plugin argues that purpose-built ML dashboards obscure that Kubernetes layer instead of exposing it. Data scientists get a UI for submitting notebooks, pipelines, and experiments. Operators and SREs ask a different set of questions: why a notebook is stuck (an ImagePullBackOff, an OOMKilled event, or a pod waiting on a PersistentVolumeClaim), which Run resources failed recently across namespaces, which parameter set a Katib Experiment reports as optimal, or whether a TrainJob resource references the TrainingRuntime it expects. Answering those questions has generally meant leaving the ML dashboard and falling back to kubectl, often across several commands, to reconstruct what is actually happening at the pod level.

What the Plugin Actually Shows

The Headlamp Kubeflow plugin closes that gap by surfacing Kubeflow’s CRDs directly inside Headlamp, the general-purpose Kubernetes web UI maintained under Kubernetes SIG UI and licensed under Apache 2.0. Headlamp runs as a desktop application or in-cluster, and its plugin architecture lets contributors add first-class views for custom resources, which is what the Kubeflow plugin does across five component families: Notebooks, Pipelines, Katib, Training, and Spark.

Notebooks

The Notebook detail view shows Pod conditions along with their reason and message fields; CPU, memory, and GPU requests and limits; volume mounts and their backing type, whether that is a PersistentVolumeClaim, ConfigMap, Secret, or emptyDir; environment variables that reference a Secret or ConfigMap; sidecar containers; and node tolerations. That consolidates information an operator would otherwise have to piece together from several separate kubectl describe commands.

Hyperparameter tuning with Katib

The Katib views expose the tuning algorithm, the search space, every Trial with its live status, and the current best Trial with its metric values and parameter assignments, along with the early-stopping configuration and how many Trials stopped early. An operator can follow a running hyperparameter search without leaving the cluster UI.

Pipelines without the backend database

The Pipelines views read Kubernetes API resources directly rather than querying the Kubeflow Pipelines API service or its backend database, so operators can inspect stored pipeline state even when that service is unavailable. The Pipeline detail view renders a side-by-side YAML diff between the latest and previous PipelineVersion, Run views show state and duration, RecurringRun views turn the underlying schedule into human-readable text, and an artifacts view aggregates pipelineRoot values from recent Run resources.

Training and Spark workloads

The plugin also covers TrainJob, TrainingRuntime, and ClusterTrainingRuntime resources for distributed PyTorch and TensorFlow training jobs, plus SparkApplication and ScheduledSparkApplication resources for large-scale data processing with Apache Spark.

A map view for ML resources

Headlamp’s map source feature renders Notebook, Profile, PodDefault, Experiment, Pipeline, SparkApplication, and TrainJob resources as graph nodes, drawing edges between them based on their ownerReferences, with inline summaries when an operator hovers over a node.

Modular by Design

Kubeflow is rarely installed as a single monolith. Teams commonly adopt only the pieces they need, running Katib without the full pipeline stack, for example, or Notebooks without Spark. The Headlamp plugin matches that reality: it discovers which Kubeflow API groups actually exist on a cluster and displays only the corresponding UI sections. The plugin’s own README, maintained in the headlamp-k8s/plugins repository, describes the same behavior from the installation side: if a team installs only Katib or only Notebooks through Helm or Kustomize, the plugin recognizes what is available and unlocks precisely those sections in the Headlamp sidebar, without requiring the entire Kubeflow platform.

A Pattern Bigger Than Kubeflow

The announcement frames the release as more than a single integration. In the Kubernetes blog post, the plugin is described as “a worked example of a pattern any CRD-heavy platform can follow: meet operators where they already work, and show them the cluster-level truth.” That framing matters because Kubeflow is not the first CRD-heavy system Headlamp has plugged into. Kubernetes SIG UI has already shipped Headlamp plugins for Karpenter, Cluster API, Volcano, and Knative, each following the same template of exposing a platform’s custom resources through a general Kubernetes UI instead of asking operators to learn another standalone tool.

Getting Started

The Kubeflow plugin’s README documents installation against an existing modular Kubeflow deployment, including example kustomize commands for adding Notebooks, Katib, Training Operators, or Kubeflow Pipelines piece by piece from the upstream kubeflow/manifests repository. It also documents a lighter evaluation path: because the plugin activates its UI sections based on which CRDs are registered, contributors can create a local kind cluster and apply only the relevant CRDs, without deploying the full Kubeflow controllers, to explore the interface at near-zero cluster cost. Because Headlamp uses a plugin architecture rather than a fork, an existing Headlamp installation, desktop or in-cluster, only needs the Kubeflow plugin added to pick up the new views.

What to Watch Next

The plugin’s real test is maintenance, not adoption. Headlamp’s plugin catalog already spans Karpenter, Cluster API, Volcano, and Knative, and each addition raises the bar for keeping pace with the CRDs it wraps: a plugin that reads custom resources directly from the API server has to track upstream schema changes in each of those projects to avoid drifting out of sync. The more interesting question sits a level above Kubeflow. If other CRD-heavy AI platforms follow the same template instead of shipping their own bespoke operator dashboard, Headlamp’s role shifts from one more Kubernetes UI option to the default second screen operators reach for whenever a platform’s own dashboard cannot explain what the cluster is actually doing.

Sources

• Kubernetes Blog: Operating AI/ML Workloads on Kubernetes, A Headlamp Plugin for Kubeflow
• GitHub: headlamp-k8s/plugins, Kubeflow Plugin for Headlamp README
• GitHub: kubernetes-sigs/headlamp
• Kubeflow: The Machine Learning Toolkit for Kubernetes

Featured image: Close-up of code on a laptop screen, representing the kind of Kubernetes tooling development behind the new Headlamp Kubeflow plugin. Photo by Marc Mueller, CC0 1.0 Universal (CC0) via Wikimedia Commons; cropped, resized, and converted to WebP.