How to Build Kubernetes Pod Networking From Scratch With Linux Network Namespaces

Every Kubernetes cluster hides a strange fact in plain sight: Kubernetes itself cannot move a single network packet. It schedules pods, watches their health, and writes state to etcd, but the actual job of wiring up a network so pods can talk to each other is delegated entirely to an external plugin called a Container Network Interface, or CNI. Most engineers run kubectl apply -f calico.yaml once, watch their nodes turn Ready, and never think about it again. This tutorial takes the opposite approach. Instead of installing a CNI, you will build what one does by hand, using the same raw Linux primitives (network namespaces, virtual Ethernet cables, and a software switch) that every real CNI is built on top of, so the next time a pod cannot reach another pod, you will actually understand why.

This is a systems tutorial, not a Kubernetes tutorial: you will not install Kubernetes, kubeadm, or any container runtime. Everything here runs directly on a single Linux machine using the ip command. Every command shown below was executed on a live Ubuntu Linux host while writing this guide, including a full simulation of two separate machines failing to reach each other and then being fixed, so the output you see is real output, not paraphrased from documentation. This tutorial was inspired by, and independently verified against, a freeCodeCamp deep dive on building CNI networking by hand, cross-checked with the official Kubernetes Network Plugins documentation, the Linux ip-netns(8) and veth(4) manual pages, the kernel’s own IP sysctl documentation, and Cilium’s own documentation on how it uses eBPF.

What You Will Build, and Why It Matters

By the end of this tutorial, you will have personally done, with plain Linux commands, the three jobs that every CNI plugin does automatically on every node in a cluster:

1. Give every pod its own isolated slice of the network stack, and a virtual cable into it. You will do this with a network namespace and a veth pair.
2. Let every pod on the same node reach every other pod on that node, without wiring a direct cable between every possible pair. You will do this with a Linux bridge, a software switch.
3. Let pods on different nodes reach each other, which is where most of the “magic” people never question actually lives. You will deliberately break this, watch the exact failure, and then fix it with routing.

Once you have done all three by hand, tools like Calico, Flannel, and Cilium stop being black boxes. They are the same three jobs, automated at cluster scale and hardened for production.

Prerequisites

• A Linux machine you can freely experiment on: a spare VM, a cloud instance, or a virtual machine on your laptop. A modern distribution such as Ubuntu 22.04 LTS, Ubuntu 24.04 LTS, or Debian 12 works well. You do not need a special kernel version; network namespaces have been a stable part of the Linux kernel for well over a decade.
• Root access, or sudo. Creating network namespaces and virtual interfaces requires the CAP_NET_ADMIN and CAP_SYS_ADMIN capabilities, which in practice means running these commands as root.
• The iproute2 package, specifically the ip command. It ships by default on virtually every modern Linux distribution; confirm it is present with ip -V.
• Basic comfort with IP addresses, subnets written in CIDR notation (like 10.0.0.0/24), and the word “gateway.” You do not need to be a network engineer, and you do not need any prior Kubernetes experience.
• No Kubernetes, Docker, or container runtime installed. That is intentional. This tutorial goes underneath all of that on purpose.

A safety note before you start: a few commands below (creating bridges, enabling packet forwarding) are things you would not want to do carelessly on a shared production machine. This tutorial shows you two ways to work: a simple version that changes your machine’s real network configuration directly (best on a disposable VM or lab box you do not depend on), and a fully sandboxed version using extra namespaces as stand-ins, which cannot affect your real host at all even if you make a mistake. That sandboxed technique is explained in Step 4, and it is exactly how the multi-machine section of this tutorial was tested.

The Concept: What a CNI Actually Does

Here is the uncomfortable truth most Kubernetes users never confront: if you bootstrap a brand-new cluster with kubeadm and immediately run kubectl get nodes, every node reports NotReady, even with a healthy control plane and a running etcd. Describing the node tells you exactly why: the Ready condition’s reason comes back as KubeletNotReady, with a message of container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized. Kubernetes will not consider a node ready until an external CNI plugin shows up and takes responsibility for the packets it refuses to touch itself. This requirement is documented directly in the Kubernetes project’s own Network Plugins reference.

A CNI plugin is just a program that the kubelet (the agent running on every node) calls every time a pod is created or destroyed, with one job: wire up that pod’s networking. Under the hood, nearly every CNI, from the simplest to the most sophisticated, is really doing three things, which map directly onto the three sections of this tutorial:

• Isolation and a cable. Each pod gets its own network namespace (so its own IP, its own routing table, its own iptables rules) and a virtual Ethernet cable connecting that namespace back to the node.
• Local switching. On a single node, all of those virtual cables plug into a shared virtual switch, a Linux bridge, so pods on the same node can reach each other.
• Cross-node delivery. Somehow, a packet leaving a pod on node A has to find its way to a pod on node B. This is the part that breaks by default, and it is where CNIs start to differ wildly in sophistication (static routes, VXLAN overlays, BGP, eBPF).

You are about to build all three, in order.

Step 1: Create Two Isolated Network Namespaces

A network namespace is a Linux kernel feature that gives a process its own private copy of the entire network stack: its own loopback interface, its own set of network interfaces, its own routing table, and its own firewall rules. By default a fresh network namespace is completely cut off, it cannot reach the host’s network, and the host cannot reach it, until you deliberately wire it up. This is the exact mechanism a container runtime uses to give every pod its own network identity.

Create two of them, named red and blue:

# Context: any modern Linux distribution with iproute2, run as root or via sudo.
sudo ip netns add red
sudo ip netns add blue
sudo ip netns list

You should see both red and blue listed (the order is not guaranteed). At this point, both namespaces exist but are completely empty and disconnected, they do not even have a loopback interface brought up yet. Try running sudo ip netns exec red ip addr and you will see only a single interface, lo, sitting in the DOWN state. That is the isolation working exactly as intended.

Common mistake: forgetting that namespaces persist after your shell session ends. They are not tied to a process; they live until you explicitly delete them with ip netns del. Leftover namespaces from earlier experiments are a common source of confusing “why does this IP already exist” errors, so it is worth running ip netns list before you start any new experiment.

Step 2: Wire Them Together With a veth Pair

A namespace on its own cannot talk to anything. To connect two namespaces, Linux provides a veth pair: two virtual network interfaces that are always created together and always act like the two ends of a single patch cable. Whatever comes in one end immediately comes out the other, even if the two ends live in completely different namespaces. This is the fundamental building block underneath almost every container networking setup.

Create a veth pair and move one end into each namespace:

# Create the virtual cable: veth-red and veth-blue are the two ends of one pair.
sudo ip link add veth-red type veth peer name veth-blue

# Move each end into its namespace.
sudo ip link set veth-red netns red
sudo ip link set veth-blue netns blue

# Assign an IP address to each end, and bring both the veth interface
# AND the namespace's own loopback interface up.
sudo ip netns exec red bash -c "ip addr add 10.0.0.1/24 dev veth-red; ip link set veth-red up; ip link set lo up"
sudo ip netns exec blue bash -c "ip addr add 10.0.0.2/24 dev veth-blue; ip link set veth-blue up; ip link set lo up"

Now test the connection from inside the red namespace:

sudo ip netns exec red ping -c 3 10.0.0.2

Real output captured while writing this tutorial:

PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.318 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.047 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=0.046 ms

--- 10.0.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2083ms
rtt min/avg/max/mdev = 0.046/0.137/0.318/0.127 ms

Zero packet loss confirms the two namespaces, which have no other route to each other, are talking exclusively through the virtual cable you just created. This is precisely the primitive a CNI plugin creates for every pod: one end of a veth pair goes into the pod’s namespace, and the other end stays on the node.

Common mistake: forgetting to bring up the namespace’s own lo (loopback) interface. A brand-new namespace starts with lo administratively down. Many tools and libraries silently depend on loopback working, so skipping this step causes confusing failures that have nothing to do with your veth pair.

Clean up before moving on so the next step starts fresh:

sudo ip netns del red
sudo ip netns del blue

Deleting a namespace automatically destroys every interface that lives inside it, including its half of any veth pair, which in turn destroys the peer end too. You do not need to delete veth interfaces separately.

Step 3: Scale Out on One Node With a Linux Bridge

A single veth pair only connects two namespaces. A real node runs many pods on the same machine, and wiring a direct cable between every possible pair would not scale (that is n(n-1)/2 cables for n pods). Instead, Linux provides a bridge, a virtual switch that forwards traffic between whatever interfaces are plugged into it, based on their MAC addresses, exactly like a physical Ethernet switch.

Build a three-namespace setup, all plugged into one bridge:

# Create the virtual switch.
sudo ip link add br-lab type bridge
sudo ip link set br-lab up

# Create three pod namespaces.
sudo ip netns add red
sudo ip netns add blue
sudo ip netns add green

# For each namespace, create a veth pair: one end goes into the namespace,
# the other end becomes a "port" on the bridge.
sudo ip link add veth-red type veth peer name veth-red-br
sudo ip link add veth-blue type veth peer name veth-blue-br
sudo ip link add veth-green type veth peer name veth-green-br

sudo ip link set veth-red netns red
sudo ip link set veth-blue netns blue
sudo ip link set veth-green netns green

# Attach the host-side ends to the bridge as ports, and bring them up.
sudo ip link set veth-red-br master br-lab
sudo ip link set veth-blue-br master br-lab
sudo ip link set veth-green-br master br-lab
sudo ip link set veth-red-br up
sudo ip link set veth-blue-br up
sudo ip link set veth-green-br up

# Give each pod an IP in the SAME subnet, since they now share one virtual switch.
sudo ip netns exec red bash -c "ip addr add 10.0.1.1/24 dev veth-red; ip link set veth-red up; ip link set lo up"
sudo ip netns exec blue bash -c "ip addr add 10.0.1.2/24 dev veth-blue; ip link set veth-blue up; ip link set lo up"
sudo ip netns exec green bash -c "ip addr add 10.0.1.3/24 dev veth-green; ip link set veth-green up; ip link set lo up"

Confirm all three ports show up on the bridge, then test connectivity from red to both of the others:

ip -br link show master br-lab
sudo ip netns exec red ping -c 2 10.0.1.2
sudo ip netns exec red ping -c 2 10.0.1.3

Real output captured while writing this tutorial (red to blue, then red to green):

PING 10.0.1.2 (10.0.1.2) 56(84) bytes of data.
64 bytes from 10.0.1.2: icmp_seq=1 ttl=64 time=0.152 ms
64 bytes from 10.0.1.2: icmp_seq=2 ttl=64 time=0.058 ms

--- 10.0.1.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.058/0.105/0.152/0.047 ms

PING 10.0.1.3 (10.0.1.3) 56(84) bytes of data.
64 bytes from 10.0.1.3: icmp_seq=1 ttl=64 time=0.280 ms
64 bytes from 10.0.1.3: icmp_seq=2 ttl=64 time=0.056 ms

--- 10.0.1.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.056/0.168/0.280/0.112 ms

All three namespaces can now reach each other through one shared bridge, using only three cables instead of three point-to-point links. This is exactly what happens on every node in a real Kubernetes cluster: every pod gets a veth pair, and the node-side ends all plug into one bridge (or an equivalent virtual switch inside the CNI’s own dataplane).

Verify before moving on: if a ping fails here, check three things in order: that the bridge itself is administratively up (ip link show br-lab should show UP), that each host-side veth end shows as a bridge port (ip -br link show master br-lab should list all three), and that you remembered to bring up lo inside each namespace.

Clean up before the next step:

sudo ip netns del red
sudo ip netns del blue
sudo ip netns del green
sudo ip link del br-lab

Step 4: Hit the Wall, the Multi-Node Boundary Problem

Everything so far has lived on one machine. Real clusters span multiple physical or virtual machines, and this is where most people’s mental model of “the network just works” quietly falls apart. By default, a node has no idea that another node’s pod subnet even exists, let alone how to reach it.

Building Two Stand-In Nodes on One Machine

If you have two real VMs on the same network, you can follow this section literally: run the “node A” commands on the first VM and the “node B” commands on the second, using each VM’s real network interface as the link between them. If you only have one machine available, you do not need a second VM to see the real failure and the real fix. Network namespaces have their own independent routing table, their own independent firewall state, and critically, their own independent copy of kernel settings like IP forwarding. That means you can safely build two fully isolated “stand-in nodes” as namespaces on a single machine, without ever touching your real host’s network configuration. This is exactly the technique used to test this section of the tutorial, entirely inside disposable namespaces on a shared machine, verified before and after to leave zero trace on the host.

Set up two stand-in nodes, vm1 and vm2, connected by a veth pair standing in for a physical network link between them, each running its own bridge and its own pod:

# Two "node" namespaces, connected by a veth pair standing in for a real network link.
sudo ip netns add vm1
sudo ip netns add vm2
sudo ip link add veth-vm1 type veth peer name veth-vm2
sudo ip link set veth-vm1 netns vm1
sudo ip link set veth-vm2 netns vm2
sudo ip netns exec vm1 bash -c "ip addr add 172.16.0.1/24 dev veth-vm1; ip link set veth-vm1 up; ip link set lo up"
sudo ip netns exec vm2 bash -c "ip addr add 172.16.0.2/24 dev veth-vm2; ip link set veth-vm2 up; ip link set lo up"

# vm1 gets its own bridge, its own pod subnet (10.0.1.0/24), and one pod.
sudo ip netns add vm1-red
sudo ip netns exec vm1 ip link add br0 type bridge
sudo ip netns exec vm1 ip addr add 10.0.1.254/24 dev br0
sudo ip netns exec vm1 ip link set br0 up
sudo ip link add veth-red type veth peer name veth-red-h
sudo ip link set veth-red netns vm1-red
sudo ip link set veth-red-h netns vm1
sudo ip netns exec vm1 ip link set veth-red-h master br0
sudo ip netns exec vm1 ip link set veth-red-h up
sudo ip netns exec vm1-red bash -c "ip addr add 10.0.1.1/24 dev veth-red; ip link set veth-red up; ip link set lo up; ip route add default via 10.0.1.254"

# vm2 gets its own bridge, its own pod subnet (10.0.2.0/24), and one pod.
sudo ip netns add vm2-blue
sudo ip netns exec vm2 ip link add br0 type bridge
sudo ip netns exec vm2 ip addr add 10.0.2.254/24 dev br0
sudo ip netns exec vm2 ip link set br0 up
sudo ip link add veth-blue type veth peer name veth-blue-h
sudo ip link set veth-blue netns vm2-blue
sudo ip link set veth-blue-h netns vm2
sudo ip netns exec vm2 ip link set veth-blue-h master br0
sudo ip netns exec vm2 ip link set veth-blue-h up
sudo ip netns exec vm2-blue bash -c "ip addr add 10.0.2.1/24 dev veth-blue; ip link set veth-blue up; ip link set lo up; ip route add default via 10.0.2.254"

First, confirm the two “nodes” can reach each other over their shared link, the way two real machines on the same network segment can:

sudo ip netns exec vm1 ping -c 2 172.16.0.2

That should succeed with 0% packet loss; node-to-node connectivity was never the problem. Now try the thing that actually matters: a pod on vm1 reaching a pod on vm2.

Watching the Packet Get Dropped

sudo ip netns exec vm1-red ping -c 2 10.0.2.1

Real output captured while writing this tutorial:

PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.
From 10.0.1.254 icmp_seq=1 Destination Net Unreachable
From 10.0.1.254 icmp_seq=2 Destination Net Unreachable

--- 10.0.2.1 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1019ms

Read that error carefully: it comes back from 10.0.1.254, which is vm1‘s own bridge gateway, not from anywhere near vm2. The packet never even left node A. The pod’s default route sent it to its node’s gateway as expected, but the node itself looked at its routing table, found no entry for 10.0.2.0/24, and gave up immediately. This is “the network just works” story breaking in the most literal sense: nobody ever told vm1 that a pod subnet called 10.0.2.0/24 exists anywhere, let alone how to reach it.

Step 5: Fix It With IP Forwarding and Static Routes

Fixing this by hand requires two separate changes on each node, and it is worth understanding exactly what each one does.

The first is IP forwarding. By default, a Linux host will only accept packets addressed to itself; it will not relay a packet that arrives on one interface and is addressed somewhere else, even if it knows how to reach the destination. This is a sensible default: an ordinary machine should not silently act as a router for other people’s traffic. The kernel’s own IP sysctl documentation defines the setting plainly: net.ipv4.ip_forward is a boolean, disabled by default, and when enabled it allows the kernel to “forward packets between interfaces.” Turning it on is what turns a plain host into a router.

The second is a static route: an explicit instruction telling the kernel “packets for this specific subnet should go out this interface, via this next hop.”

Apply both fixes, scoped only to the two stand-in node namespaces:

# Turn each stand-in node into a router.
sudo ip netns exec vm1 sysctl -w net.ipv4.ip_forward=1
sudo ip netns exec vm2 sysctl -w net.ipv4.ip_forward=1

# Tell each node explicitly how to reach the OTHER node's pod subnet.
sudo ip netns exec vm1 ip route add 10.0.2.0/24 via 172.16.0.2
sudo ip netns exec vm2 ip route add 10.0.1.0/24 via 172.16.0.1

Now repeat the exact same test from before:

sudo ip netns exec vm1-red ping -c 3 10.0.2.1

Real output captured while writing this tutorial:

PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.
64 bytes from 10.0.2.1: icmp_seq=1 ttl=62 time=0.329 ms
64 bytes from 10.0.2.1: icmp_seq=2 ttl=62 time=0.095 ms
64 bytes from 10.0.2.1: icmp_seq=3 ttl=62 time=0.098 ms

--- 10.0.2.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2058ms
rtt min/avg/max/mdev = 0.095/0.174/0.329/0.109 ms

Notice the TTL dropped from 64 to 62 compared to the single-node tests earlier: the packet is now genuinely crossing two extra router hops (vm1’s bridge-to-uplink hop, then vm2’s uplink-to-bridge hop) to get there, exactly as you would expect from real inter-node routing. You can see the route you added by listing vm1‘s routing table:

sudo ip netns exec vm1 ip route show

10.0.1.0/24 dev br0 proto kernel scope link src 10.0.1.254
10.0.2.0/24 via 172.16.0.2 dev veth-vm1
172.16.0.0/24 dev veth-vm1 proto kernel scope link src 172.16.0.1

That middle line is the fix: anything addressed to 10.0.2.0/24 (vm2’s pod subnet) goes out through veth-vm1, via 172.16.0.2 (vm2’s real address on the shared link).

Common mistake: adding the route on only one node. Ping needs a working path in both directions: the request has to reach vm2, and vm2’s reply has to find its way back to vm1. If you only add the route on vm1, the request arrives at vm2 but the reply has nowhere to go, and the ping still times out. This is a frequent source of confusion when troubleshooting real asymmetric-routing bugs in production clusters, not just in this lab.

This “manual static route on every node” trick is not just a teaching exercise: it is a real, production-used CNI strategy. Flannel’s host-gw backend, for example, works by watching the Kubernetes API for every node’s assigned pod CIDR and real IP address, then programming exactly this kind of route on every other node automatically. You just did by hand, for two nodes, what that backend does at cluster scale.

Clean up everything from this step (this destroys all four namespaces and everything inside them):

sudo ip netns del vm1
sudo ip netns del vm2
sudo ip netns del vm1-red
sudo ip netns del vm2-blue

Step 6: Why Cloud CNIs Reach for Overlays and eBPF

The static-route trick you just did has a hard limitation: it only works when every node can add a route directly to every other node, which generally means all the nodes sit on the same Layer 2 network or a network you fully control the routing table for. That assumption breaks constantly in the real world. Cloud VPCs often restrict how many custom routes you can add, span multiple availability zones or regions with their own routing domains, or simply do not let tenants inject arbitrary routes into the underlying fabric at all.

This is why more sophisticated CNIs like Cilium reach for two other techniques. The first is an overlay network: instead of asking the physical network to route pod-subnet traffic directly, the CNI wraps each pod packet inside an ordinary packet addressed node-to-node (commonly using VXLAN or Geneve encapsulation), sends it across whatever network already exists between the nodes, and unwraps it on arrival. From the physical network’s point of view, it just sees normal traffic between two known node IPs; the pod-to-pod addressing is hidden inside. Cilium’s own documentation describes this directly, offering “overlay networking: encapsulation-based virtual network spanning all hosts with support for VXLAN and Geneve,” explicitly because it “works on almost any network.”

The second technique is eBPF, a Linux kernel technology that lets a program attach directly inside the kernel’s own packet-processing path, instead of relying on the traditional iptables rule chains you may have used to fix the routing in Step 5. Cilium describes eBPF as the foundation that lets it insert “powerful security visibility and control logic within Linux itself.” In practice this means Cilium can enforce network policy and forward packets with far less per-packet overhead than a long chain of iptables rules, which matters enormously once a cluster has thousands of pods and network policies.

Strip away the marketing language from any CNI’s architecture diagram and you will find the same three jobs you just built by hand: isolate and cable each pod, switch between pods on the same node, and get packets to pods on other nodes, however cleverly that last part is implemented.

Verifying Everything Works End to End

Before you consider this tutorial complete, confirm you can explain, in your own words, each of these checkpoints, and reproduce them if asked:

• Isolation: a fresh network namespace (ip netns add) starts with no working connectivity at all, not even loopback, until you explicitly bring interfaces up.
• The cable: a veth pair (ip link add ... type veth peer name ...) connects exactly two namespaces, and destroying either end destroys both.
• The switch: a Linux bridge (ip link add ... type bridge) lets any number of veth pairs reach each other by attaching their host-side ends as ports (ip link set ... master).
• The wall: without a route, a different node has no way to know a remote pod subnet exists, and fails fast with “Destination Net Unreachable” rather than hanging.
• The fix: enabling net.ipv4.ip_forward and adding an explicit ip route on both sides restores connectivity, and the dropped TTL confirms the packet is truly taking extra hops.

If any step did not match the output shown here, work backward from the last thing that worked: check ip netns list for stale namespaces, ip -br link show (or ip -br link show master <bridge>) for interfaces stuck administratively down, and ip route show inside the relevant namespace for a missing or mistyped route.

Cleaning Up

If you followed along step by step and cleaned up at the end of each section as instructed, there should be nothing left to remove. As a final sanity check, confirm no lab namespaces or bridges remain:

ip netns list
ip -br link show

The first command should print nothing, and the second should show only your machine’s normal interfaces (things like lo, your real Ethernet interface, and anything Docker or your distribution created on its own). Leftover lab namespaces are harmless but easy to forget about; a stray ip netns del <name> or ip link del <name> clears any stragglers.

Next Steps

Now that you have built these primitives with your own hands, the most valuable next step is to go find them again inside a real cluster. Install kubeadm and a real CNI such as Calico or Cilium on a disposable VM (or a set of them), then, on a worker node, run ip netns list, bridge link (or ip -br link show), and ip route show. You will find the CNI’s own version of the namespaces, veth pairs, bridges (or their eBPF equivalents), and routes you just built manually, often with recognizable naming patterns specific to that CNI. From there, the official Kubernetes Cluster Networking documentation and Cilium’s own architecture documentation are the natural next reads, now backed by a mental model you built yourself instead of one you took on faith.