Understanding the Kubernetes architecture

Introduction

One cool thing about Kubernetes is that the infrastructure is transparently handled by Kubernetes objects themselves. If you want to see how everything works together, you can just inspect the pods, services and other objects themselves.

Disclaimer: This setup is valid for Kubernetes in Docker Desktop on Mac. In a managed environment, or in your setup, the setup might be different and components are not necessarily deployed as simple pods.

Let’s have a first look. I’m using the built-in Kubernetes for Docker Desktop on a Mac here (and use k as an alias to kubectl)

1╰$ k get all
2NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
3service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   39d

Well, not a lot going on here. Is that all there is?

Of course not, the real meat is in other namespaces. Anyway, let’s first inspect this service we found:

1╰$ k describe service/kubernetes
2Name:              kubernetes
3Namespace:         default
4Labels:            component=apiserver
5                   provider=kubernetes
6Annotations:       <none>
7Selector:          <none>
8Type:              ClusterIP
9IP:                10.96.0.1
10Port:              https  443/TCP
11TargetPort:        6443/TCP
12Endpoints:         192.168.65.3:6443
13Session Affinity:  None
14Events:            <none>

What is it doing? It’s actually just forwarding requests to the Kubernetes API server (which we will meet soon).

Okay, but where is the rest?

Let’s check out the other namespaces.

1╰$ k get namespaces
2NAME              STATUS   AGE
3default           Active   39d
4kube-node-lease   Active   39d
5kube-public       Active   39d
6kube-system       Active   39d

The default namespace we already know.

kube-node-lease is relatively new (starting from version 1.14) and helps in determining the availability of a node via heartbeats. We’ll ignore it for now.

kube-public is a namespace that exists if the cluster was created with kudeadm. It contains a single, lonely configmap called config-info and aids in discovery for other clients. Read more about the discovery API if you are interested.

kube-system is the most interesting namespace for us. Let’s dive in:

The kube-system namespace

Let’s do a quick lookup:

1╰$ k get all -n kube-system
2NAME                                         READY   STATUS    RESTARTS   AGE
3pod/coredns-f9fd979d6-dmkxf                  1/1     Running   0          4m57s
4pod/coredns-f9fd979d6-vrd8t                  1/1     Running   0          4m57s
5pod/etcd-docker-desktop                      1/1     Running   0          3m43s
6pod/kube-apiserver-docker-desktop            1/1     Running   0          4m7s
7pod/kube-controller-manager-docker-desktop   1/1     Running   0          3m45s
8pod/kube-proxy-s4jfp                         1/1     Running   0          4m57s
9pod/kube-scheduler-docker-desktop            1/1     Running   0          3m51s
10pod/storage-provisioner                      1/1     Running   0          3m43s
11pod/vpnkit-controller                        1/1     Running   0          3m42s
12
13NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
14service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   5m4s
15
16NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
17daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   5m4s
18
19NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
20deployment.apps/coredns   2/2     2            2           5m4s
21
22NAME                                DESIRED   CURRENT   READY   AGE
23replicaset.apps/coredns-f9fd979d6   2         2         2       4m57s

By the way: kubectl get all does not really get all resource types, only the most important. Check out ketall if you want to get everything.

A lot more going on here! We can see see a deployment (plus replicaset and its pods) of coredns and a service kube-dns, which, unsurprisingly, handle DNS in the cluster. I will skip this for now and write a separate post about DNS in Kubernetes.

So, what is left? I count 7 pods, let’s go through them one by one.

VPN Controller

pod/vpnkit-controller is there to interact with my host’s VPN configuration. Not very interesting for us right now.

etcd

etcd is a key-value store that stores information concerning the cluster, like pods, nodes, roles, configs, secrets. Think of it as its database. Every piece of information you get via kubectl get comes from etcd and every update you make goes into etcd. As Kubernetes is a distributed system the database needs to be distributed, too, and etcd is the perfect solution for it.

API server

kube-apiserver is quite self-explanatory. This is the endpoint with which other components interact with REST operations if they want to lookup or change state. If you run kubectl you are interacting with the apiserver.

It’s also the only component that interacts with etcd.

Controller Manager

kube-controller-manager. Controllers (e.g. Node controllers, deployment controllers, namespace controllers) supervise the state of the components they are in charge of. Node controllers, for example, check the health of nodes every five seconds to see if they are still reachable. If they don’t respond after 40 seconds it is marked unreachable. If it doesn’t come up again after 5 minutes it evicts the pods scheduled on it to other nodes.

All these controllers run in the process of the controller-manager.

Kube Proxy

kube-proxy runs on every node and handles service networking. As a service is not a real object but just lives in memory, somebody needs to handle the routing of services and kube-proxy does just this. One way it does this is with iptable rules. On every node it creates entries for every service and forwards traffic to the IP of the actual pod selecting the service.

Kube Scheduler

kube-scheduler decides which pod gets scheduled on which node. Any pod might have specific resource requirements to run.

In the first phase the scheduler filters out the nodes which can’t satisfy the resource requirements or are tainted or don’t tolerate this specific pod.

Then it ranks the remaining nodes by a priority function. If the pod is CPU-intensive the node with more free CPU capacity gets a higher rank and so one. The one with the best ranking wins the pod.

Storage Provisioner

storage-provisioner handles storage specific to the platform being used. In my case this pod is using the docker/desktop-storage-provisioner image specific for Mac’s Docker Desktop.

By the way, you can see more of the configuration for some of these by looking at the config maps in the kube-system namespace

1╰$ k get cm -n kube-system
2NAME                                 DATA   AGE
3coredns                              1      6m5s
4extension-apiserver-authentication   6      6m8s
5kube-proxy                           2      6m5s
6kubeadm-config                       2      6m6s
7kubelet-config-1.19                  1      6m6s

Conclusion

That’s it, a quick primer on what is going on under the hood of a Kubernetes setup. While Kubernetes does a ton of things and in the end is quite complex, the infrastructure is very modular and understandable. It’s possible to do a deep dive into any component you are interested it, if you so desire.