Kubernetes’ controller-runtime Cache Shows Why Controller Reads Never Touch the API Server

Ask a Go developer how r.Get() works inside a Kubernetes controller and most will say it queries the API server directly. A Kubernetes Blog post published July 29, 2026 by Ænix’s Andrei Kvapil and Timofei Larkin argues that assumption is backwards, and that getting it wrong is the root cause of a recurring class of production incidents: controllers that quietly balloon in memory, silently degrade into O(n) scans, or trip over stale reads seconds after a write.

The correction matters because controller-runtime is not a niche library. It is the foundation kubebuilder scaffolds on top of, and the base most serious Go operators on Kubernetes are built from, including the growing wave of Kubernetes-native AI agent and platform controllers. Its core claim: r.Get() and r.List() inside a reconciler almost never touch kube-apiserver. They read from a local, in-memory cache that a manager process builds once at startup and keeps current through a watch. Writes, Create, Update, Patch, Delete, are the exception: those still go straight to the API server. That split, cheap reads from memory paired with exact writes to etcd, is not a shortcut bolted onto Kubernetes for performance. It is how the control plane has worked since the first kube-controller-manager, and controller-runtime just wraps it in a friendlier interface.

Why Kubernetes Never Learned to Poll

The alternative to this design is easy to picture: a dozen controllers in a cluster, each firing a get and a list per reconcile, at hundreds of reconciles per second. The API server and etcd would fall over in minutes. Kubernetes was built around a watch model instead of polling from its earliest controllers. A client issues list once, gets a snapshot of the objects it cares about along with a resourceVersion, then opens a watch starting from that version and keeps a local copy current over a single long-lived HTTP connection. There is no repeated “what does the world look like right now” loop.

This pattern has lived in client-go since the earliest versions of kube-controller-manager. controller-runtime does not reinvent it; it wraps the same primitives, a Reflector, a DeltaFIFO queue, and an Indexer, in a framework so individual controller authors do not have to wire them together by hand.

Inside the Pipeline: Reflector, DeltaFIFO, and the Indexer

The package at sigs.k8s.io/controller-runtime/pkg/cache is a thin wrapper over k8s.io/client-go/tools/cache. Four pieces do the actual work.

The Reflector Owns the Only Connection to the API Server

The Reflector is the single component that talks to kube-apiserver. It performs one list at startup, then keeps a watch open from the resourceVersion that list returned. If the connection drops, it reconnects from the last known version. Only if the API server responds 410 Gone, meaning that version has aged out of etcd’s history, does the Reflector fall back to a fresh list. Kubernetes calls this a relist, and outside of that failure path it does not happen on any schedule.

DeltaFIFO Orders Events, It Does Not Collapse Them

DeltaFIFO is the buffer between the Reflector and the rest of the informer. Its job is narrower than it sounds: preserve the order of events per object key, and group everything that happened to a given namespace/name into one slice so a consumer sees the full history since its last read. Its own source shows the deduplication is narrow too: a helper called dedupDeltas only collapses consecutive Deleted events for the same key. Consecutive Added or Updated deltas are left alone.

That has a concrete effect. Create a Pod and, within a second or two, the scheduler assigns a node, the kubelet marks it Pending, then ContainerCreating, then Running. Every one of those updates fires the registered event handler separately; DeltaFIFO does not merge them into one call. What actually keeps a controller from reconciling five times in a row is a second, separate layer: the workqueue. Event handlers extract a namespace/name key and enqueue it, and re-enqueuing an existing key silently coalesces with the entry already sitting in the queue. By the time a worker pops that key and Reconcile runs, the informer’s store already holds the final state, so Reconcile only fires once even though five separate delta events led up to it.

The Indexer Is Just a Map

Underneath, the Indexer (also called ThreadSafeStore) is a map[string]interface{} keyed by namespace/name, a mutex, and a dictionary of any registered indexes. No B-tree, no LSM tree, nothing exotic. That is exactly why a cache-hit Get costs microseconds: it is a map lookup and a struct copy, not a network round trip.

One Informer, Many Subscribers

The keyword in SharedIndexInformer is shared. A manager creates exactly one informer per GroupVersionKind, and every controller, webhook, and event source registered against that manager reads from and subscribes to the same store. From the API server’s point of view, that is one list and one watch per type, no matter how many reconcilers live inside the process.

Where the Mental Model Breaks in Practice

Because the manager always finishes its initial list and watch setup, calls cache.WaitForCacheSync, and only then lets workers start draining the queue, a controller can never observe “running but the cache is still empty.” Even the very first Get in a reconciler’s lifetime reads from a fully warmed, fully indexed store. That guarantee is also where most of the surprises start.

• Expecting read-after-write. Call r.Update(), then immediately call r.Get() on the same object, and the second read can still return the pre-update value. This is not a bug; it is optimistic concurrency doing its job. Every write carries the resourceVersion the client last saw, and the API server rejects a write with 409 Conflict if etcd already has a newer version. Between a successful update landing in etcd and the watch event refreshing the local cache, there is a real window, typically milliseconds, where a read returns stale data. The fix is not to add a delay and re-check. It is to write Reconcile so it is idempotent and safe to run again if the world has not caught up yet.
• Mutating shared objects. Objects handed to a Predicate or an EventHandler are the literal objects sitting in the informer’s shared store, not a copy. controller-runtime deep-copies on Get and List, so those results are safe to mutate freely, but editing a label or a status field directly inside a predicate or handler corrupts the shared cache for every other controller subscribed to that type. If a predicate or map function sets a field on the object it was handed, that is a bug waiting to surface somewhere else in the process.
• Treating resync as relist. Every informer carries a resyncPeriod, ten hours by default in controller-runtime (confirmed directly in the cache package source). A resync does not perform a new list. It re-emits every object already sitting in the store as a synthetic Sync event, giving a controller that somehow missed an update a chance to reconcile it again, at zero cost to the API server.
• Reaching for a timer instead of RequeueAfter. Reconcilers that need to wait and retry should return ctrl.Result{RequeueAfter: d} rather than spinning up time.Sleep or a goroutine. The request goes back into the deduplicated workqueue with a delayed trigger, and if a real event arrives for the same object before the timer fires, the reconcile runs immediately instead of waiting it out.

Turning the Cache Into a Query Engine

An unindexed List filtered in a loop works fine until a cluster has tens of thousands of objects and a controller is scanning that whole set on every reconcile. The field indexer built into controller-runtime turns the cache into something closer to a queryable store. Registering an index at manager startup:

if err := mgr.GetFieldIndexer().IndexField(
    ctx,
    &corev1.Pod{},
    "spec.nodeName",
    func(obj client.Object) []string {
        pod := obj.(*corev1.Pod)
        if pod.Spec.NodeName == "" {
            return nil
        }
        return []string{pod.Spec.NodeName}
    },
); err != nil {
    return err
}

builds an inverted index (a dictionary from field value to the set of matching object keys) that stays current as objects change, so a later query does no scanning at all:

var pods corev1.PodList
_ = r.List(ctx, &pods,
    client.MatchingFields{"spec.nodeName": "node-1"},
)

The comparison to SQL holds up better than it sounds:

Three caveats keep this from being magic. MatchingFields only works for fields that already have a registered IndexField; nothing is inferred automatically. It supports equality only, no ranges or sorts (a common workaround for time-based queries is to index a rounded, bucketed timestamp instead of the raw value). And MatchingLabels is not indexed at all: ThreadSafeStore keeps no separate label dictionary, so a label-selector query still walks every cached object of that type.

Keeping the Cache From Eating Your Memory

By default an informer pulls every object of its type across every namespace, which is a multi-gigabyte surprise the moment it touches Secrets, Nodes, or Events in a large cluster. Helm alone can turn Secrets into a memory problem, since it stores release state as helm.sh/release.v1.* secrets that often run to about a hundred kilobytes each.

controller-runtime exposes a cache.Options struct (verified against the current source, which defines exactly these fields) for narrowing what gets cached, passed when constructing the manager:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    Cache: cache.Options{
        ByObject: map[client.Object]cache.ByObject{
            &corev1.Secret{}: {
                Namespaces: map[string]cache.Config{
                    "my-controller": {},
                },
            },
            &corev1.Pod{}: {
                Transform: func(obj any) (any, error) {
                    pod := obj.(*corev1.Pod)
                    pod.ManagedFields = nil
                    return pod, nil
                },
            },
        },
    },
})

Namespaces and label or field selectors become parameters of the watch itself, so the API server only ever sends matching objects. Transform runs before an object lands in the store, a good place to strip managedFields or other bulk a controller does not need. One caveat worth flagging in a review: this is a manager-level setting, so narrowing the cache for a type affects every controller in that process, not just the one that asked for it.

For cases where only an object’s existence or metadata matters, not its spec or data, PartialObjectMetadata caches just ObjectMeta (labels, annotations, owner references, finalizers). For Secrets specifically, that can cut memory for that type by an order of magnitude.

When the Cache Genuinely Isn’t Enough

mgr.GetAPIReader() returns a reader that bypasses the cache and hits the API server directly. It earns its keep in a handful of specific situations: validating webhooks where a stale read could wrongly block a legitimate update, one-off reads of a type the controller does not otherwise watch, any read that happens before mgr.Start() runs (the regular client returns nothing useful that early), and paginated traversal with client.Limit and client.Continue, since the cache-backed client always returns its full local result set and ignores pagination parameters.

For a more permanent case, client.CacheOptions.DisableFor (also confirmed directly against the client package source) turns caching off for a type entirely, so no informer starts and no list runs at startup for it:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    Client: client.Options{
        Cache: &client.CacheOptions{
            DisableFor: []client.Object{
                &corev1.Secret{},
            },
        },
    },
})

The one pattern to avoid regardless: reaching into the cache and falling back to a live API call when it misses. That reintroduces exactly the split-brain risk the cache design exists to avoid, a local copy that believes one thing while the API server has already moved on.

A Short Checklist Before Shipping a Controller

• Constrain cache scope for “fat” types (Secret, ConfigMap, Event, Pod, Node) with Namespaces, label, or field selectors.
• Add a Transform to strip fields like managedFields that a controller never reads.
• Register an IndexField for every field a List call filters on with MatchingFields; without one, the query fails outright rather than silently scanning.
• Call DeepCopy before mutating anything received in a Predicate or EventHandler.
• Write Reconcile to be idempotent, since it must behave correctly whether it fires once or five times in a row for the same change.
• Reach for APIReader, not the regular client, anywhere freshness genuinely matters: webhooks, initialization, one-off reads.
• Use PartialObjectMetadata for types where only metadata is needed.
• Never call mgr.GetClient() before mgr.Start() finishes syncing.
• Use RequeueAfter for deferred work instead of time.Sleep or a hand-rolled goroutine.

None of this is a knock on the design of controller-runtime. Reading from memory and writing straight to etcd is precisely what lets a cluster run a dozen custom controllers, security operators, GitOps controllers, and the growing pile of Kubernetes-native AI agent substrates, without each one hammering the API server on every reconcile. The risk sits entirely on the human side: kubebuilder makes it possible to scaffold a working controller in an afternoon without ever being told that its reads are eventually consistent by design. As Kvapil and Larkin put it in the original post, once the fact that Get never calls the API server becomes a reflex, half the questions that come up in controller code reviews answer themselves.