Well , what’s so special about networking in kubernetes apart from other networking implementations ? That’s because k8s don’t have one. To be precise k8s don’t implement networking natively, it depends on networking plugins which are not part of the standard k8s objects.

Anyone can implement networking in k8s by implementing the container network interface (CNI). CNI is a set of standards to be followed in order to establish pod to pod communication in the same node or different nodes. These standards are created so that everyone can implement their own network solutions but adhering to a standard which makes it compatible to work with k8s. Let us have a brief about how pods communicate with each other.

The thumb rule of k8s networking is that each pod has it’s own unique ip throughout the cluster thus avoiding the overhead of doing NAT. Whenever a pod is getting created it is getting placed in it’s own network namespace (more about network namespaces here). And there is a default host network namespace that comes with the linux installation. Each network namespace has one or more network interface attached to them with the IP address assigned to it.

The interfaces that are being attached to the pods are actually a pair of interfaces (veth pair) in which one of the interface is attached to the host network namespace as shown below.

so we have two network interfaces in the same host and how do we connect them now? That’s done via a virtual bridge in the host netns. so whenever a pod sends an IP packet destined to a pod in the same host ,it spits it out to the other veth pair connected to root netns and the bridge do the ARP request to all the interfaces connected to it and sends the packet to the pod accordingly.

So, how does the pods on different hosts communicate with each other ? There are multiple ways of implementing it. Each CNI plugin implements this in it’s own way , cloud plugins like AWS VPC CNI when you have a cluster in eks , do it simply by attaching their Elastic network interface to the host whenever a pod is getting created and pod gets the IP which is from the VPC subnet. Now, since the Elastic interfaces are in the same subnet they can talk to each other like normal hosts do.

But in a non cloud environment , an Overlay network (Logical network formed across different hosts) is created by the CNI plugin like Flannel,Cilium or Calico to achieve pod to pod communication. Overlay networks needs a separate discussion. so we will talk about it in later posts.

So now we know who creates all this (CNI plugin of course) and one such CNI plugin is Calico.

All networking plugins implements both networking (routing traffic) and network security (filtering traffic), so does calico. You can install calico in your k8s cluster following this doc.

Calico creates all these components under the respective namespaces.

pod/tigera-operator = is responsible for installing the CRDs (Custom resource definitions) that makes calico as a kubernetes object and also responsible to bring up the calico controllers (policy,ns,pod,node controllers) and apply changes to calico resources.

pod/calico-kube-controllers — is responsible for recognising changes in Kubernetes objects that impact routing. The controller contains multiple controllers inside of it, watching changes in network policies, services, namespaces, service accounts, nodes.

pod/calico-node — core of calico , runs as a daemon set and have three internal daemons running that are responsible for different networking functions.

Felix — is responsible for programming the host to satisfy pod routes and network policy. To do this, it interacts with the Linux kernel’s route tables and the Linux IPtables (for network policy).

Bird — takes routing rules written by Felix and peers with other BIRD instances that run on all hosts by default. These BGP peers are constantly sharing routing information about their known routes. BIRD is a capable daemon that enables a multitude of topologies.

Confd — An open source project that manage local application configuration files using templates and data from etcd.

pod/calico-typha — Typha is a process that fans out configuration to all calico-node instances in a cluster. It acts as a cache that can de-duplicate events from the API server, lessening load significantly. As the Calico datastore changes, these changes must be propagated to every instance of calico-node, which can be hundreds or thousands of instances. This creates scaling issues in a cluster with more than 50 nodes as every node will be opening up a watch for API server events. Typha gets around this by being an intermediary between the API server and all instances of calico-node.

pod/calico-apiserver — Install the Calico API server on an existing cluster to enable management of Calico APIs using kubectl instead of using calicoctl.

The Calico Datastore is a generic term referring to whatever is used to store Calico configuration, routing, policy, and other information. Calico supports 2 datastore modes, Kubernetes and etcd.

So, this is how it goes. when you create a calico object say ,a Network policy the request will reach the kube-apiserver via the calico-apiserver which informs one of the calico kube controller responsible for policy management. The controller puts the data in the datastore and typha fans out this new configuration to all the calico-node deamonset. The felix process then updates the IP table accordingly and if it’s a route table update, then the BIRD daemon will share the route with other bird peers in the nodes across the cluster.

You can use calico to enforce only the network policy and let the aws vpc cni plugin to handle the networking part. To do this calico runs in felix only mode by default when installed in managed cloud services.

Happy Networking :)