0. What progress have I made?#

Since my last post, I’ve had plenty of time to properly study Kubernetes from a book that attempts to cover the subject comprehensively, and I must say the difference from online guides was noticeable. After some research, I decided to go with O’Reilly’s Kubernetes Up & Running, 3rd Edition because I found it offered a practical approach to Kubernetes with real-world examples.

Both during my studying process and after that, with what I felt to be some solid basis on the matter, I decided to test my knowledge with yet another test cluster.

1. The test setup(s)#

Being a really impatient person, I could not stand properly testing k8s until I had full knowledge of it before adopting it in my homelab, so I ended up with a semi-production environment made out of my two mini-PCs in which I used the configs I wrote and experimented with in my true test-environment that was virtualized on Proxmox.
About that, at the time I did not have a Proxmox server but thankfully my IT manager at E-AgleTRT gave me permission to create three Ubuntu Server VMs on the team cluster to use as a playground since someone with k8s proficiency in the organization was necessary anyway.

These two setups were quite different both in the number and type of nodes and in the approach used:

1.1 Virtualized environment#

As mentioned above I had three nodes: I used the less powerful one (4C-8GB RAM) as the master and the beefier two (8C-8GB RAM) as the two workers.
Having PBS backups, I did not hesitate to try new things such as Patroni’s StatefulSets or NFS and SMB backend storage servers. I still need to test the CEPH storage backend that would be useful for the Formula Student team’s future cluster.

Here I tried both PVCs and Storage Classes definition and usage, autoscaling functionalities, Deployments and Service types and generally did lots of troubleshooting since nothing was working most of the time.

1.2 Physical environment#

In this second setup, I hosted every service I used, so it needed to be way more stable, and the total absence of the security granted by PBS backups had me thinking and experimenting a lot in the VMs before modifying the deployment configurations.

I managed to keep this cluster up and running the whole time since its creation and I admit that all the pressure derived from it having all my data and services probably taught me how to operate in production environments. Also, data and config backups were quite a challenge in this case since I wanted them to be managed by the master and not by an external device, but more on that can be found in the following sections.

2. Volumes and persistent data#

Working with services that relied on pod lifecycle-persistent data, I needed to save configs and user information in some type of storage accessible from every node in the cluster; fortunately, k8s provides volumes exactly for this use case.

In the previous post, I explained how I “hadn’t properly considered persistent volumes,” which basically means that I used hostPath volumes—meaning that the container inside the pod was actually mounting and operating on a folder in the node’s filesystem.
While this approach can work while using a one-node cluster, it becomes completely unsustainable the second a second node is added. This is due to the fact that, not knowing where a pod will be scheduled, it is good practice to ensure that every node has complete access to every volume that a pod relies on, and with local paths, this is a pain.

The first thing I thought was to create an NFS share with all the main’s local paths that were used by the pods and then to mount them in the exact location on every node, but this is far from a standard approach and I wanted to learn how to properly handle volumes so I discarded this option.
Knowing that my own cluster only had two nodes, I knew I couldn’t use solutions such as GlusterFS or CEPH that require a minimum of three nodes to work properly with a quorum system, so I decided to stick with NFS since for my use case it was more than enough.

3. Deployments and StatefulSets#

During my previous tests, I only used the deployment type for all my pods since it was the most used one until one day, while trying to host a Postgres database on my cluster, I stumbled across DaemonSets, StatefulSets, and ReplicaSets.
They all have key differences and knowing them helped me to better deploy all the containers that I needed both for testing purposes and for my own convenience.

I will briefly revise the key concepts present in the official k8s documentation of the mainly used ones.

  • StatefulSets: Used to manage pod-lifecycle persistent applications (such as the Postgres instance in my example). It creates a given number of pods, each with its identity (pod-0, pod-1, pod-2, …, pod-n). This is very helpful to manage per-pod persistent storage or network identity.

  • ReplicaSets: Used to achieve stable pod presence in the cluster. Given a number of pods for a specific application, it works to keep the real cluster state as the expected one by dynamically creating and deleting pods as the cluster’s condition changes.

  • Deployments: Used to manage ReplicaSets; given a desired state, the Deployment Controller changes the ReplicaSet’s actual state to the desired state at a controlled rate. Deployments can also be used to dynamically create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments.

Having that covered, in my own cluster I used mainly StatefulSets since the only thing I care about is data persistence and unique pod IDs, while I don’t need to distribute pods due to my cluster’s low number of nodes and each node’s ability to perfectly handle the workload of each application.
I took advantage of the second test environment to properly use Deployments and almost every functionality that they offer (I kept auto-scaling for the future) and I managed to host multiple instances of OwnCloud, AdGuard, and Apache.

4. Networking#

Kubernetes networking can be, as all its other functionalities, a bit overwhelming compared with Docker, but it offers excellent scalability and general adaptability to every use case.

Every pod’s port must be exposed through a Service that allows the creation of:

  • ClusterIP: to expose the service on a cluster-internal IP that, as the name suggests, can be accessed only within the cluster.
  • NodePort: to expose the service on each cluster’s node’s IP at a specific port. This may create conflicts with other services already running on some or all the cluster’s nodes.
  • LoadBalancer: to expose the service using an externally provided load balancer, useful when working with cloud providers.
  • ExternalName: used to map a service to a DNS name.

I mainly used LoadBalancers and ClusterIP (for Postgres, Redis, and some self-hosted APIs).

5. Conclusions#

I am currently continuing to experiment with Kubernetes, knowing that there are topics that I still need to study or apply, but I am happy with the knowledge level I am achieving and the fact that I have completely embraced k8s in my homelab. Until now, it has not let me down once.
I am overall having a great time and I hope to keep going with this project and these updates.

Thank you for your attention.