How to Build and Deploy a Custom Prometheus Metrics Exporter for Kubernetes

Kubernetes only sees two signals out of the box: CPU and memory. That is enough to keep a web server from falling over, but it tells you nothing about how many jobs are waiting in a queue, how many WebSocket connections a pod is holding, or how long the last batch run took. This tutorial builds the piece that closes that gap: a custom Prometheus metrics exporter, written in Go, containerized, deployed to a real Kubernetes cluster, and verified end to end so Prometheus is actually scraping your application’s own numbers. Every command and code sample below is drawn from and cross-checked against the official Kubernetes Blog walkthrough, the Kubernetes HorizontalPodAutoscaler documentation, and the official Prometheus documentation.

What a Metrics Exporter Actually Does, and Why You Would Build One

Prometheus works by pulling, not by being pushed to. On a regular interval, called the scrape interval, the Prometheus server sends an HTTP request to a /metrics path on every target it knows about and reads back a plain text response: one line per metric, with a name, optional key and value labels, and a number. A metrics exporter is nothing more than a small HTTP server whose only job is to produce that text. Where the numbers come from is entirely up to you: a database query, a message queue’s depth, an internal counter your application already tracks.

If you own the application’s source code, you can usually skip a separate exporter and expose /metrics directly from inside the same process using a Prometheus client library. A standalone exporter earns its keep when the data lives outside your application (a third-party service, a piece of hardware, a database you cannot modify) or when you specifically do not want to touch the target application’s code.

Understanding Metric Types Before You Write Any Code

The Prometheus data model defines three metric types you will use constantly. Picking the right one matters because using the wrong type produces numbers that lie about what they measure:

• Counter: a value that only ever goes up (or resets to zero when the process restarts). Correct for totals like requests served or jobs completed. Never use a counter for something that can go down, such as the number of items currently in a queue.
• Gauge: a value that can freely rise and fall. Correct for point-in-time snapshots: queue depth, active connections, memory in use.
• Histogram: records how many observations fall into configurable buckets, so you can calculate percentiles like p50 or p99 later instead of only an average. Correct for measuring durations and sizes, such as how long a request took.

These definitions match the Prometheus project’s own documentation. Name your metrics <namespace>_<name>_<unit> in snake_case, for example worker_queue_depth or worker_job_duration_seconds. A clear name saves the next person, often you, at 2 a.m., from having to guess what a graph is actually showing.

Prerequisites

• Go 1.21 or newer installed locally (go version to check). The examples below use the standard library plus the official Prometheus Go client.
• Docker Engine 24+ or Podman 4+ to build a container image. Any OCI-compatible builder works; the commands below use Docker syntax.
• kubectl, configured against a real Kubernetes cluster you can deploy to (a local cluster such as kind or minikube is fine).
• Prometheus already running in the cluster, ideally installed through the Prometheus Operator (commonly via the kube-prometheus-stack Helm chart). This tutorial assumes Prometheus itself is already installed; installing Prometheus from scratch is its own separate task.
• A container registry you can push to (Docker Hub, GHCR, or a registry local to your cluster).
• Assumed prior knowledge: comfort running basic kubectl commands (apply, get, describe) and reading YAML. Go experience helps but is not required; every line of code below is explained.

Step 1: Set Up the Go Project

Create a new directory and initialize it as a Go module. A Go module is simply a project folder with a go.mod file that tracks its name and dependencies:

# Context: any machine with Go 1.21+ installed, terminal open.
mkdir my-exporter && cd my-exporter
go mod init example.com/my-exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

What this does and why it matters: go mod init example.com/my-exporter creates the module. example.com is a placeholder domain reserved for documentation and examples, so replace it with your own path, such as github.com/yourname/my-exporter, in a real project. The two go get commands pull in the official Prometheus Go client: prometheus gives you the metric types from the previous section, and promhttp gives you a ready-made HTTP handler that serializes them into the text format Prometheus expects.

Step 2: Register Your Metrics

Create a file named main.go. Before a metric can be updated, it has to be declared and registered so the client library knows it exists:

# Context: my-exporter/main.go, new file, first block.
package main

import (
    "log"
    "net/http"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    jobsProcessed = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "worker_jobs_processed_total",
            Help: "Total number of jobs processed, partitioned by status.",
        },
        []string{"status"},
    )

    queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "worker_queue_depth",
        Help: "Current number of jobs waiting in the queue.",
    })

    jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
        Name:    "worker_job_duration_seconds",
        Help:    "Time spent processing a single job.",
        Buckets: prometheus.DefBuckets,
    })
)

func init() {
    prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration)
}

What this does and why it matters: each New... call creates one metric of the type discussed above: a CounterVec (a counter with a status label, so you get separate totals for success and failure without writing separate metrics), a plain Gauge for queue depth, and a Histogram with Prometheus’s default bucket boundaries for job duration. prometheus.MustRegister in init() adds all three to Prometheus’s default registry, which is what makes them show up in the exporter’s output at all; forgetting to register a metric is a common mistake covered later in this tutorial. MustRegister panics immediately if you accidentally register the same metric name twice, which is a feature: it turns a configuration mistake into a startup crash you cannot miss, instead of a silent bug you find weeks later.

Step 3: Collect Real Values With a Polling Loop

Registered metrics start at zero and stay there until something updates them. The pattern below runs in its own goroutine, a lightweight concurrent function that Go can run alongside the rest of your program, and periodically refreshes each metric from whatever your application actually tracks:

# Context: my-exporter/main.go, appended below Step 2's code.
import (
    "math/rand"
    "time"
)

func collectMetrics() {
    for {
        // Replace this block with real reads from your database,
        // internal API, or message queue.
        depth := float64(rand.Intn(50))
        queueDepth.Set(depth)

        start := time.Now()
        time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond)
        jobDuration.Observe(time.Since(start).Seconds())
        jobsProcessed.WithLabelValues("success").Inc()

        time.Sleep(5 * time.Second)
    }
}

What this does and why it matters: queueDepth.Set(...) overwrites the gauge with a fresh snapshot, which is exactly what a gauge is for. jobDuration.Observe(...) records one duration sample into the histogram’s buckets, and jobsProcessed.WithLabelValues("success").Inc() increments the counter’s status="success" series by one. The five-second sleep at the end of the loop is the polling interval; keep it shorter than Prometheus’s scrape interval (fifteen seconds is the common default) so that every scrape sees a value that was refreshed recently rather than one that is about to go stale.

Step 4: Expose the Metrics and Health Endpoints

With the metrics registered and being updated, the last piece is an HTTP server to serve them. Add a main function:

# Context: my-exporter/main.go, appended below Step 3's code.
func main() {
    go collectMetrics()

    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })

    log.Println("Listening on :8080")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatalf("server error: %v", err)
    }
}

What this does and why it matters: go collectMetrics() starts the polling loop from Step 3 in the background so it does not block the HTTP server from starting. promhttp.Handler() is the piece that turns your registered metrics into the Prometheus text exposition format on every request to /metrics; you never have to format that text by hand. The separate /healthz path returns a bare 200 with no metric data, which gives Kubernetes a lightweight target for liveness probes in Step 7 without mixing health checks and metrics traffic on the same response.

Step 5: Run and Verify the Exporter Locally

Before packaging anything into a container, confirm it works as a plain local process:

# Context: my-exporter/, all four steps saved in main.go.
go run .

Expected output: a single log line, Listening on :8080, with no errors. Leave that terminal running and, in a second terminal, request the metrics endpoint:

# Context: second terminal, same machine.
curl http://localhost:8080/metrics | grep worker_

Expected output: lines beginning with # HELP worker_... and # TYPE worker_... for all three metrics, followed by their current values, for example worker_queue_depth 23. If those lines are missing, double check that collectMetrics is actually being started with go collectMetrics() in main, and that the metric variable names in Step 2 match the ones you are updating in Step 3 exactly.

Step 6: Package the Exporter as a Container Image

Kubernetes runs containers, not local Go binaries, so the next step is a Dockerfile. A multi-stage build compiles the binary in one throwaway stage and copies only the finished binary into a minimal final image, so the shipped image does not carry a full Go toolchain:

# Context: my-exporter/Dockerfile, new file.
FROM golang:1.21-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /exporter .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /exporter /exporter
EXPOSE 8080
ENTRYPOINT ["/exporter"]

What this does and why it matters: the first stage uses the official golang:1.21-alpine image to compile a statically linked binary (CGO_ENABLED=0 disables C bindings so the binary has no external library dependencies). The second stage starts over from gcr.io/distroless/static:nonroot, a Google-maintained base image that contains no shell, no package manager, and runs as a non-root user by default, and copies in only the compiled binary. The result is a small, low-attack-surface image that satisfies most cluster security policies without extra configuration. Build and push it, replacing <registry> with your own registry path:

# Context: my-exporter/, Dockerfile saved, logged in to your registry.
docker build -t <registry>/my-exporter:v1.0.0 .
docker push <registry>/my-exporter:v1.0.0

Running these two commands by hand is fine for this tutorial; in a real project, wire them into a CI/CD pipeline instead so every image is built the same way, every time.

Step 7: Deploy the Exporter to Kubernetes

Two manifests are enough: a Deployment to keep the pod running, and a Service so Prometheus has a stable address to scrape. Both examples below use a monitoring namespace, a common convention for running Prometheus and related components together; adjust it to match your own cluster:

# Context: my-exporter/deployment.yaml, new file.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: my-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: my-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: my-exporter
    spec:
      containers:
      - name: exporter
        image: <registry>/my-exporter:v1.0.0
        ports:
        - name: metrics
          containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          requests:
            cpu: 50m
            memory: 32Mi
          limits:
            cpu: 100m
            memory: 64Mi

# Context: my-exporter/service.yaml, new file.
apiVersion: v1
kind: Service
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: my-exporter
spec:
  selector:
    app.kubernetes.io/name: my-exporter
  ports:
  - name: metrics
    port: 8080
    targetPort: metrics

What this does and why it matters: the Deployment’s livenessProbe points at /healthz from Step 4, so Kubernetes restarts the pod automatically if the process stops responding, without that check ever touching the metrics data itself. The conservative resources block is appropriate for a lightweight, single-purpose process like this exporter. The Service names its port metrics, a name the ServiceMonitor in the next step references directly instead of a raw port number, which keeps the configuration readable and lets the underlying port change without breaking the reference. Apply both files:

# Context: kubectl configured against your cluster.
kubectl apply -f deployment.yaml -f service.yaml
kubectl get pods -n monitoring -l app.kubernetes.io/name=my-exporter

Expected output: a pod named my-exporter-... with STATUS Running and READY 1/1. If it instead shows ImagePullBackOff, double check that the image path in deployment.yaml matches exactly what you pushed in Step 6, including the registry and tag.

Step 8: Tell Prometheus Where to Scrape

How you configure scraping depends on how Prometheus itself was installed.

Option 1: Prometheus Operator (ServiceMonitor)

If Prometheus was installed with the Prometheus Operator or the kube-prometheus-stack Helm chart, create a ServiceMonitor. Its release label has to match the label selector configured on your Prometheus resource; kube-prometheus-stack is the default label value for a standard Helm install:

# Context: my-exporter/servicemonitor.yaml, new file. Requires the Prometheus Operator CRDs already installed.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-exporter
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

# Context: kubectl configured against your cluster.
kubectl apply -f servicemonitor.yaml

Option 2: Annotation-Based Discovery

If your Prometheus server uses annotation-based pod discovery instead of the Operator, confirm with whoever manages Prometheus that a matching scrape_config rule already exists, then add these annotations to the pod template in deployment.yaml:

# Context: my-exporter/deployment.yaml, inside spec.template.metadata.
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"
  prometheus.io/path: "/metrics"

The Operator ignores these annotations, so it is harmless to add them even if you are not sure which setup you have. If you genuinely do not know which one your cluster uses, the ServiceMonitor path is more explicit and easier to debug, so start there.

Step 9: Verify Prometheus Is Actually Scraping Your Metric

Port-forward to the Prometheus service and open its targets page:

# Context: kubectl configured against your cluster, Prometheus running in the monitoring namespace.
kubectl port-forward svc/prometheus-operated 9090 -n monitoring

Navigate to http://localhost:9090/targets in a browser. Look for a target labeled my-exporter; its state should read UP. If it reads DOWN or does not appear at all, confirm the pod is running and that the ServiceMonitor’s release label matches your Prometheus resource:

# Context: second terminal, kubectl configured against your cluster.
kubectl get pods -n monitoring -l app.kubernetes.io/name=my-exporter
kubectl describe servicemonitor my-exporter -n monitoring

Once the target shows UP, run a query in the Prometheus expression browser (the same localhost:9090 page, under the Graph tab) to confirm data is actually flowing end to end:

rate(worker_jobs_processed_total{status="success"}[2m])

A non-zero result means the entire pipeline works: your exporter is producing data, Prometheus found it and is scraping it on schedule, and the resulting time series is stored and queryable.

Common Mistakes and How to Catch Them

• Forgetting to register a metric. A Gauge or Counter you create with prometheus.NewGauge(...) but never pass to prometheus.MustRegister(...) will silently never appear in /metrics, with no error anywhere. If a metric is missing from the scrape output, check the MustRegister call in init() first.
• Using a Counter for a value that can decrease. Queue depth goes up and down, so it has to be a Gauge. Modeling it as a Counter produces a chart with impossible drops that break any rate() calculation built on top of it.
• A mismatched ServiceMonitor release label. This fails silently: no error, the target simply never shows up on the /targets page. Confirm the label on your ServiceMonitor matches the label selector your Prometheus custom resource actually uses, which is kube-prometheus-stack only for the default Helm install name; a custom Helm release name changes it.
• A polling interval longer than the scrape interval. If collectMetrics only refreshes a value every 30 seconds but Prometheus scrapes every 15, roughly half your data points repeat the previous value. Keep the polling loop faster than the scrape interval.
• Trying to kubectl exec into a distroless container to debug it. The gcr.io/distroless/static:nonroot base image has no shell, so kubectl exec -it <pod> -- sh fails outright. Use kubectl logs for application output, and kubectl debug to attach an ephemeral debug container with its own shell if you need to poke around interactively.
• Exposing /metrics without restricting access to it. Metrics can leak internal details such as queue names, error messages, or internal hostnames. For anything beyond a local test cluster, restrict the metrics port with a NetworkPolicy so only Prometheus itself can reach it.

Verifying It Works End to End

1. go run . locally starts with no errors, and curl http://localhost:8080/metrics | grep worker_ returns all three metrics with current values.
2. docker push succeeds, and kubectl get pods -n monitoring shows the exporter pod as Running and 1/1 ready.
3. Either the ServiceMonitor or the annotation-based scrape config from Step 8 is applied and matches how your Prometheus instance is actually configured.
4. http://localhost:9090/targets (via kubectl port-forward) shows the my-exporter target in state UP.
5. The query rate(worker_jobs_processed_total{status="success"}[2m]) in the Prometheus expression browser returns a non-zero series.

If every one of those checks passes, you have a working custom metrics pipeline: an application-level signal your team defined, flowing from your own code into Prometheus on a schedule you control.

Next Steps: Wiring This Into Autoscaling

A working exporter is the foundation, not the destination. The natural next step is feeding worker_queue_depth into a HorizontalPodAutoscaler so your workload scales on the signal that actually reflects load, instead of only CPU. That requires a metrics adapter, most commonly the Prometheus Adapter, which registers Prometheus metrics with the custom.metrics.k8s.io API so the autoscaler can read them. Once the adapter is configured, a Pods-type metric block that scales on the exact gauge this tutorial built looks like this, using the autoscaling/v2 API (stable since Kubernetes 1.23) per the official walkthrough:

# Context: HorizontalPodAutoscaler using autoscaling/v2, referencing a metric already registered with the Prometheus Adapter.
metrics:
- type: Pods
  pods:
    metric:
      name: worker_queue_depth
    target:
      type: AverageValue
      averageValue: "10"

That single block tells the HorizontalPodAutoscaler to add replicas whenever the average queue depth across pods climbs past 10, on top of, or instead of, the CPU-based scaling most tutorials stop at. Setting up the Prometheus Adapter itself, including its scrape-query mapping rules, is enough material for a tutorial of its own; the two links above are the right place to continue from here.

Sources

• Kubernetes Blog: Building a Custom Metrics Exporter for Kubernetes
• Kubernetes Documentation: HorizontalPodAutoscaler Walkthrough
• Prometheus Documentation: Metric Types
• kubernetes-sigs/prometheus-adapter (official GitHub repository)
• prometheus/client_golang (official GitHub repository)
• Featured image source: NASA/JPL-Caltech, Deep Space Network operations, Wikimedia Commons

Image credit: Blanca Renteria, Artemis Deep Space Network operations chief, monitors real-time data at NASA’s Jet Propulsion Laboratory Space Flight Operations Facility. NASA/JPL-Caltech, public domain; cropped, resized, and converted to WebP for sxz.io.