The machines
Four bare Debian 12 machines — no cloud account, no installer, no operator. The jumpbox runs everything with 512MB of RAM.
| Machine | CPU | RAM | Storage | Role |
|---|---|---|---|---|
| jumpbox | 1 | 512MB | 10GB | administration base |
| server | 1 | 2GB | 20GB | control plane |
| node-0 | 1 | 2GB | 20GB | worker · 10.200.0.0/24 |
| node-1 | 1 | 2GB | 20GB | worker · 10.200.1.0/24 |
$ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
All binaries, downloaded once
The jumpbox gets the CLI tools (wget curl vim openssl git), a clone of the course repo, and every binary the cluster will run — 500+ MB, downloaded once, sorted into client/ controller/ worker/ cni-plugins/, then copied to the other machines over SSH.
$ wget -q --show-progress \
-i downloads-$(dpkg --print-architecture).txt
$ kubectl version --client Client Version: v1.32.3
machines.txt is the whole inventory
One line per machine. This plain text file is the cluster’s entire inventory system — every later lab greps it or while-read loops over it: SSH keys, hostnames, a shared /etc/hosts.
XXX.XXX.XXX.XXX server.kubernetes.local server XXX.XXX.XXX.XXX node-0.kubernetes.local node-0 10.200.0.0/24 XXX.XXX.XXX.XXX node-1.kubernetes.local node-1 10.200.1.0/24
$ while read IP FQDN HOST SUBNET; do
ssh-copy-id root@${IP}
done < machines.txt
$ while read IP FQDN HOST SUBNET; do ssh -n root@${HOST} hostname; done < machines.txt server node-0 node-1
One CA, eight certificates
You become your own certificate authority: ca.key + ca.crt, RSA 4096, ten years. One bash for-loop over an 8-element array creates every identity in the cluster from ca.conf sections.
subject = CN=system:node:node-0, O=system:nodes
The certificate is the identity. A node’s Kubernetes username and group live in the cert subject; the Node Authorizer reads them from the TLS handshake.
$ openssl x509 -in node-0.crt -noout -subject subject=CN = system:node:node-0, O = system:nodes
Cluster + credentials + context, in one file
Six kubeconfigs, one per client, all --embed-certs=true so each file is self-contained. Every kubeconfig points at server.kubernetes.local:6443 — except admin, which uses 127.0.0.1:6443 because it runs on the server itself.
$ kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt --embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=node-0.kubeconfig
$ ls *.kubeconfig admin.kubeconfig kube-scheduler.kubeconfig kube-controller-manager.kubeconfig node-0.kubeconfig kube-proxy.kubeconfig node-1.kubeconfig
A 32-byte key encrypts Secrets at rest
head -c 32 /dev/urandom | base64 generates key1 for the aescbc provider (with identity fallback). The key goes into an EncryptionConfiguration the API server loads at startup — from then on, Secrets are written to etcd as ciphertext.
$ export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64) $ envsubst < configs/encryption-config.yaml \ > encryption-config.yaml
$ grep -A1 'name: key1' encryption-config.yaml - name: key1 secret: 1oR4hQXUnFhKW6uRLKPyFyVvxrHiPrqPbHKI3EX7g1o=
etcd starts
The first process to run. Kubernetes components are stateless — etcd is the single source of cluster state, one member on the server. chmod 700 /var/lib/etcd, and the kube-api-server cert pair is reused for etcd’s TLS.
$ etcdctl member list 6702b0a34e2cfd39, started, controller, http://127.0.0.1:2380, http://127.0.0.1:2379, false
The control plane comes up
Three systemd units land on the server — kube-apiserver, kube-controller-manager, kube-scheduler — fed by everything staged so far: certs, kubeconfigs, key1, etcd. Then an RBAC ClusterRole (system:kube-apiserver-to-kubelet) lets the API server reach the kubelets — that’s what makes logs, exec and metrics work.
$ curl --cacert ca.crt https://server.kubernetes.local:6443/version
{
"major": "1",
"minor": "32",
"gitVersion": "v1.32.3",
"goVersion": "go1.23.6",
"platform": "linux/arm64"
}
TLS signed by your own CA. From here on, every kubectl command goes through this endpoint.
$ kubectl cluster-info --kubeconfig admin.kubeconfig Kubernetes control plane is running at https://127.0.0.1:6443
runc → CNI → containerd → kubelet → kube-proxy
Each worker gets the full container stack plus OS prep: swapoff -a (the kubelet refuses to start with swap on by default), the br-netfilter module with bridge-nf-call-iptables=1, and socat conntrack ipset kmod — socat is what makes kubectl port-forward work in lab 12. One sed stamps each node’s identity into the network config: the literal word SUBNET becomes 10.200.0.0/24 on node-0 and 10.200.1.0/24 on node-1.
$ SUBNET=$(grep node-0 machines.txt | cut -d " " -f 4) $ sed "s|SUBNET|${SUBNET}|g" configs/10-bridge.conf \ > 10-bridge.conf
$ systemctl enable --now containerd kubelet kube-proxy
$ kubectl get nodes --kubeconfig admin.kubeconfig NAME STATUS ROLES AGE VERSION node-0 Ready <none> 35s v1.32.3 node-1 Ready <none> 10s v1.32.3
kubectl works from the jumpbox
The same four kubectl config commands as lab 5, without --kubeconfig — so the result lands in the default ~/.kube/config. The endpoint server.kubernetes.local:6443 resolves through the /etc/hosts entries from lab 3.
$ kubectl config set-cluster kubernetes-the-hard-way \ --certificate-authority=ca.crt --embed-certs=true \ --server=https://server.kubernetes.local:6443 $ kubectl config set-credentials admin \ --client-certificate=admin.crt --client-key=admin.key $ kubectl config set-context kubernetes-the-hard-way \ --cluster=kubernetes-the-hard-way --user=admin $ kubectl config use-context kubernetes-the-hard-way
$ kubectl get nodes NAME STATUS ROLES AGE VERSION node-0 Ready <none> 2m v1.32.3 node-1 Ready <none> 2m v1.32.3
Cross-node pod traffic: four static routes
Pods on different nodes can’t reach each other yet — nothing maps a pod CIDR to the node that owns it. Four ip route add commands across the three machines (IPs and subnets grepped from machines.txt) fix that. Pod networking here is plain Linux routing.
server$ ip route add 10.200.0.0/24 via ${NODE_0_IP} server$ ip route add 10.200.1.0/24 via ${NODE_1_IP} node-0$ ip route add 10.200.1.0/24 via ${NODE_1_IP} node-1$ ip route add 10.200.0.0/24 via ${NODE_0_IP}
$ ssh root@server ip route default via XXX.XXX.XXX.XXX dev ens160 10.200.0.0/24 via XXX.XXX.XXX.XXX dev ens160 10.200.1.0/24 via XXX.XXX.XXX.XXX dev ens160
Verify encryption, networking, services
Six checks against the running cluster: hexdump a Secret straight out of etcd (the prefix shows the aescbc key at work), then a Deployment, port-forward 8080→80 (that’s socat from lab 9), logs, exec, and a NodePort service.
$ etcdctl get /registry/secrets/default/kubernetes-the-hard-way | hexdump -C 00000040 6b 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 3a 76 |k8s:enc:aescbc:v| 00000050 31 3a 6b 65 79 31 3a 9b 61 3f 65 21 ad a0 8f 15 |1:key1:.a?e!....|
$ kubectl create deployment nginx --image=nginx:latest $ kubectl expose deployment nginx --port 80 --type NodePort
$ curl -I http://node-0:${NODE_PORT} HTTP/1.1 200 OK Server: nginx/1.27.4
to complete the teardown.
Nothing exists outside the four machines — no cloud resources, no external state. Everything the course created was files and processes, and both go away with the VMs. The diagram on the right clears accordingly.
$ # no teardown commands — deleting the VMs removes everything END OF COURSE · 13/13 LABS