Operating Kafka at scale with Strimzi

2025-01-19

+

+

+

+

+

+
The source code for this lab exercise is available on GitHub.

Apache Kafka (Kafka) is an open source event streaming platform governed by the vendor-neutral Apache Software Foundation. It is optimized for processing vast amounts of data in real time, focused on availability, reliability and scalability. At the heart of Kafka is the publish-subscribe (pub/sub) model, topics and messages, with producers writing (producing) messages to one or more topics and consumers reading (consuming) messages from one or more topics.

Kafka can be flexibly deployed to a wide variety of environments and is also available as a managed service by cloud service providers, e.g.:
1. Deploying on VM / bare metal directly from upstream releases
2. Deploying as containers on a single host with Docker Compose V2
3. Deploying to Kubernetes with Strimzi
4. Available as a managed service by cloud service providers such as Amazon MSK or Confluent for Kafka
Deploying on VM / bare metal requires extensive manual operation for deployment, scaling and upgrades and is difficult to manage at scale so it is not covered in this article. Leveraging a managed Kafka service alleviates the operational burden from Kafka users and developers but may introduce vendor-specific features and optimizations leading to vendor lock-in. Therefore, we’ll start with a reproducible deployment of Kafka on a single host with Docker Compose V2, then move forward to deploying Kafka on Kubernetes with Strimzi in the second half of the article and appreciate the automated deployment and lifecycle management of Kafka clusters it delivers. In a sense, Kafka on Kubernetes with Strimzi gives the best of both worlds – you get an experience similar to managed Kafka services but with the added benefits of being vendor-neutral and retaining control over your data.

Before we start, let’s take a quick overview of Kafka’s architecture.

Kafka architecture

Source: KRaft: Apache Kafka Without Zookeeper

Kafka clusters consist of controllers and brokers as illustrated by the diagram to the right. Controllers store Kafka metadata and manage the entire cluster, while brokers store messages in topic partitions and process write requests from producers as well as read requests from consumers. Kafka controllers are analogous to Kubernetes control plane nodes and brokers are analogous to worker nodes.

In the legacy Kafka architecture illustrated by the diagram to the left, Kafka clusters store their metadata in Apache Zookeeper and assign a Kafka broker as Zookeeper controller for interacting with Zookeeper. This configuration is deprecated and Zookeeper support will be removed in the upcoming 4.0 release.

Lab: Deploy Kafka to Docker and Kubernetes

This lab has been tested with Kubernetes v1.32 (Penelope).

Prerequisites

Familiarity with Linux and Kubernetes is assumed. Otherwise, check out the courses below for a quick introduction or refresher.
1. LFS101x: Introduction to Linux
2. LFS158x: Introduction to Kubernetes
Setting up your environment

A Linux environment with at least 2 vCPUs, 8GiB memory and sufficient available disk space capable of running Docker. This can be your own desktop/laptop if you’re a Linux user (like I am ;-), or a spare board (e.g. Raspberry Pi), physical server, virtual machine or cloud instance.

The reference environment is Ubuntu 24.04 LTS (Noble Numbat) so if you’re on a different Linux distribution, adapt apt-related commands with dnf / pacman / something else accordingly when installing system packages. Otherwise, the remaining instructions should be broadly applicable to most Linux distributions.

Install Docker and Docker Compose V2

We’ll use Docker for 2 purposes:
1. In the first half of the lab, we’ll use Docker with Docker Compose V2 to spin up a Kafka cluster manually with 1 controller and 1 broker
2. In the second half of the lab, we’ll use Docker to spin up a kind Kubernetes cluster and use it to spin up Kafka clusters automatically with the Strimzi operator
Install Docker and Docker Compose V2, then add the current user to the docker group:
sudo apt update sudo apt install -y docker.io docker-compose-v2 sudo usermod -aG docker "${USER}"
Log out and in for the changes to take effect.

Check that we have the correct version of Docker installed:
docker version
Sample output:
Client: Version: 26.1.3 API version: 1.45 Go version: go1.22.2 Git commit: 26.1.3-0ubuntu1~24.04.1 Built: Mon Oct 14 14:29:26 2024 OS/Arch: linux/amd64 Context: default Server: Engine: Version: 26.1.3 API version: 1.45 (minimum version 1.24) Go version: go1.22.2 Git commit: 26.1.3-0ubuntu1~24.04.1 Built: Mon Oct 14 14:29:26 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.12 GitCommit: runc: Version: 1.1.12-0ubuntu3.1 GitCommit: docker-init: Version: 0.19.0 GitCommit:
Check that the Docker Compose V2 plugin is installed:
docker compose version
Sample output:
Docker Compose version 2.27.1+ds1-0ubuntu1~24.04.1
Install kind, kubectl and Helm for Kubernetes

Now install kind, kubectl and Helm for Kubernetes. To install kind, simply follow the instructions in their Quickstart:
# For AMD64 / x86_64 [ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.26.0/kind-linux-amd64 # For ARM64 [ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.26.0/kind-linux-arm64 chmod +x ./kind sudo mv ./kind /usr/local/bin/kind
Check the correct kind version is installed:
kind version
Sample output:
kind v0.26.0 go1.23.4 linux/amd64
For kubectl, the official instructions will suffice:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/.
Check that kubectl is correctly installed:
kubectl version --client
Sample output:
Client Version: v1.32.0 Kustomize Version: v5.5.0
For command-line completion, add the following line to your ~/.bashrc:
source <(kubectl completion bash)
Now save the file and run:
source ~/.bashrc
Likewise, install Helm with the commands below:
wget https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz tar xvf helm-v3.16.4-linux-amd64.tar.gz sudo mv linux-amd64/helm /usr/local/bin/helm
Check the Helm version:
helm version
Sample output:
version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"clean", GoVersion:"go1.22.7"}
Add the following line to your ~/.bashrc for command-line completion:
source <(helm completion bash)
Now save the file and run:
source ~/.bashrc
Install cloud-provider-kind for load balancing

kubernetes-sigs/cloud-provider-kind provides load-balancing capabilities for kind Kubernetes clusters, by automatically spinning up an Envoy proxy container for each LoadBalancer service. The LoadBalancer service is accessible only from the Docker host by default. Specify the -enable-lb-port-mapping option to expose LoadBalancer services externally, which opens a random high-numbered port on the Docker host for each port specified in the LoadBalancer service.

Download and install cloud-provider-kind:
wget https://github.com/kubernetes-sigs/cloud-provider-kind/releases/download/v0.5.0/cloud-provider-kind_0.5.0_linux_amd64.tar.gz tar xvf cloud-provider-kind_0.5.0_linux_amd64.tar.gz sudo mv cloud-provider-kind /usr/local/bin/cloud-provider-kind
An example systemd unit file cloud-provider-kind.service to start and manage cloud-provider-kind as a background daemon:
# /etc/systemd/system/cloud-provider-kind.service [Unit] Description=Cloud Provider Kind Service After=network.target [Service] Type=simple ExecStart=/usr/local/bin/cloud-provider-kind -enable-lb-port-mapping Restart=on-failure [Install] WantedBy=multi-user.target
Install the unit file to /etc/systemd/system/cloud-provider-kind.service, then restart systemd, start the service and enable it to auto-start at system boot:
sudo systemctl daemon-reload sudo systemctl enable --now cloud-provider-kind.service
Deploy Kafka to Docker with Docker Compose V2

Docker Compose V2 provides a convenient method of spinning up multiple related containers on a single Docker host using a compose.yaml configuration file. If you’re familiar with Kubernetes, understanding Docker Compose should be a breeze and requires no introduction 😉

Clone the sample repository for this lab and make it your working directory: DonaldKellett/kafka-kraft-1c1b
git clone https://github.com/DonaldKellett/kafka-kraft-1c1b.git pushd kafka-kraft-1c1b/
Take a moment to inspect compose.yaml and understand the configuration, then start the Kafka cluster with Compose:
docker compose up -d
Sample output:
[+] Running 3/3 ✔ kafka-broker-0 Pulled 80.2s ✔ kafka-controller-0 Pulled 80.2s ✔ 69b14d9ee1f8 Pull complete 74.7s [+] Running 3/3 ✔ Network kafka-kraft-1c1b_app-tier Created 0.1s ✔ Container kafka-controller-0 St... 1.7s ✔ Container kafka-broker-0 Starte... 0.7s
Tip: the sample repo uses the bitnami/kafka image from Docker Hub.

Recall that Kafka messages are divided into topics. Producers write messages to a Kafka topic which can then be read by consumers from the same topic. Let’s create a new topic kafka-hello-world with the kafka-topics.sh script bundled with Kafka on our controller by connecting to our broker:
docker exec kafka-controller-0 \ /opt/bitnami/kafka/bin/kafka-topics.sh \ --bootstrap-server kafka-broker-0:9092 \ --create \ --topic kafka-hello-world
Sample output:
Created topic kafka-hello-world.
Now open a new terminal tab or window and run the kafka-console-consumer.sh script on our controller to start listening for messages in the kafka-hello-world topic. Keep the tab or window open and keep the script running – you should see no output for now.
docker exec -it kafka-controller-0 \ /opt/bitnami/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server kafka-broker-0:9092 \ --topic kafka-hello-world \ --from-beginning
Return to your original terminal tab or window and run the kafka-console-producer.sh script on our controller to start writing messages to the kafka-hello-world topic. You should see an angular prompt > within a few seconds:
docker exec -it kafka-controller-0 \ /opt/bitnami/kafka/bin/kafka-console-producer.sh \ --bootstrap-server kafka-broker-0:9092 \ --topic kafka-hello-world
Write a few messages separated by newlines. For example:
>Hello World! >This is my first Kafka cluster!
Now watch the messages appear in the consumer’s console. For example:
Hello World! This is my first Kafka cluster!
Press Ctrl+C on both terminal tabs or windows when done. You should also the following output printed to the consumer’s console:
Processed a total of 2 messages
Shut down the Kafka cluster and remove the containers:
docker compose down
Sample output:
[+] Running 3/3 ✔ Container kafka-broker-0 Remove... 1.3s ✔ Container kafka-controller-0 Re... 0.7s ✔ Network kafka-kraft-1c1b_app-tier Removed 0.2s
Before we move on to Kafka on Kubernetes with Strimzi, here are a few exercises to consider if you would like to learn more about Kafka and how it works:
1. Fork the repository and clone it (your fork) again so your modifications can be saved and published to GitHub, or another Git forge of your choice
2. What happens if we remove the dependency of the broker on the controller in compose.yaml? Try starting and shutting down the Kafka cluster a few times
3. The Kafka cluster loses all its data in its current configuration when it is shut down. Modify the compose.yaml file to introduce persistence so Kafka retains its topics and messages when it is shut down and restarted.
4. Make the control plane highly available by modifying compose.yaml to manually scale the number of controllers to 3. Why does Kafka require an odd number of controllers for high availability?
5. Make the data plane highly available by modifying compose.yaml to manually scale the number of brokers to 3 or above. Why is 3 the minimum recommended number of brokers for high availability?
Clean up the repository as we won’t need it for our next section.
popd rm -rf kafka-kraft-1c1b/
Deploy Kafka to Kubernetes with Strimzi

Strimzi is an operator for running Kafka clusters on Kubernetes. It automates the deployment and lifecycle management of Kafka clusters so users and developers do not have to worry about which environment variables to set on each Kafka server or manually migrating each node in legacy Kafka clusters with Zookeeper to the modern, native Kafka Raft (KRaft) architecture. Strimzi was originally developed by Red Hat and is now a CNCF Incubating project.

Prepare a private root CA

Let’s prepare a private root CA. This will come in handy later.

The openssl commands use to generate the certificates are explained in detail in my other blog post so they are presented without explanation here: Keycloak on Fedora Workstation with Podman | Fedora Magazine

Create our root CA certificate-key pair:
openssl req \ -x509 \ -new \ -nodes \ -keyout my-root-ca.key \ -sha256 \ -days 3650 \ -out my-root-ca.crt \ -subj '/CN=My root CA/DC=localhost'
Install the root CA certificate to /usr/local/share/ca-certificates/my-root-ca.crt and run update-ca-certificates:
sudo cp my-root-ca.crt /usr/local/share/ca-certificates/my-root-ca.crt sudo update-ca-certificates
Start your Kubernetes cluster

With kind, your Kubernetes cluster is 1 command away.
kind create cluster
Install the Kubernetes Ingress NGINX controller

We’ll expose our Strimzi Kafka cluster with Ingress. Install the ingress-nginx controller with Helm and set it as default IngressClass. We’ll need to enable TLS passthrough as well for secure communication to our Strimzi Kafka clusters via Ingress since Kafka uses a custom binary protocol:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm -n ingress-nginx install \ --create-namespace \ ingress-nginx \ ingress-nginx/ingress-nginx \ --version 4.12.0 \ --set controller.ingressClassResource.default=true \ --set controller.extraArgs.enable-ssl-passthrough=true
Get the IP of our ingress controller and prepare a TLS serving certificate

Get the LoadBalancer IP of our ingress controller with the command below:
kubectl -n ingress-nginx get svc \ ingress-nginx-controller \ -o jsonpath='{.status.loadBalancer.ingress.*.ip}'
Sample output:
172.18.0.3
Since the LoadBalancer IP is 172.18.0.3 in our case (yours might be different!), let’s create a TLS serving certificate for the IP address 172.18.0.3 and the following domains signed by our root CA:
1. strimzi-kafka.172.18.0.3.sslip.io
2. strimzi-kafka-broker-0.172.18.0.3.sslip.io
3. strimzi-kafka-broker-1.172.18.0.3.sslip.io
4. strimzi-kafka-broker-2.172.18.0.3.sslip.io
Remember to adapt the domain name and IP address based on the LoadBalancer IP address you got in the previous step.
openssl req \ -new \ -newkey rsa:2048 \ -nodes \ -keyout strimzi-kafka.key \ -sha256 \ -out strimzi-kafka.csr \ -subj '/CN=Strimzi Kafka/DC=localhost' \ -addext 'subjectAltName=DNS:strimzi-kafka.172.18.0.3.sslip.io,DNS:strimzi-kafka-broker-0.172.18.0.3.sslip.io,DNS:strimzi-kafka-broker-1.172.18.0.3.sslip.io,DNS:strimzi-kafka-broker-2.172.18.0.3.sslip.io,IP:172.18.0.3' \ -addext 'basicConstraints=critical,CA:FALSE' \ -addext 'keyUsage=digitalSignature,keyEncipherment' \ -addext 'extendedKeyUsage=serverAuth' openssl x509 \ -req \ -days 365 \ -in strimzi-kafka.csr \ -CA my-root-ca.crt \ -CAkey my-root-ca.key \ -out strimzi-kafka.crt \ -set_serial 01 \ -sha256 \ -copy_extensions copy
Deploy the Strimzi operator with Helm

Let’s deploy the Strimzi operator with Helm and configure it to watch all namespaces. The Strimzi operator is responsible for managing Strimzi and Kafka related custom resources.
helm -n strimzi-kafka-operator install \ --create-namespace \ strimzi-kafka-operator \ oci://quay.io/strimzi-helm/strimzi-kafka-operator \ --version 0.45.0 \ --set watchAnyNamespace=true
Wait for the operator to become ready.
kubectl -n strimzi-kafka-operator wait \ --for=condition=Ready \ pods \ -l name=strimzi-cluster-operator \ --timeout=180s
Sample output:
pod/strimzi-cluster-operator-76b947897f-np8t2 condition met
Deploy a Strimzi Kafka cluster with 1 controller and 1 broker

Create a namespace strimzi-kafka for our Kafka cluster:
kubectl create ns strimzi-kafka
Now create a secret strimzi-kafka-cluster-tls containing the TLS serving certificate and key we generated earlier, to be used for accessing our Strimzi Kafka cluster over TLS ingress.
kubectl -n strimzi-kafka create secret tls \ strimzi-kafka-cluster-tls \ --cert=strimzi-kafka.crt \ --key=strimzi-kafka.key
Save the YAML manifests below as strimzi-kafka-1c1b.yaml. They include:
1. A KafkaNodePool of 1 controller
2. A KafkaNodePool of 1 broker
3. A Kafka cluster consisting of both node pools above
--- apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaNodePool metadata: name: strimzi-kafka-controller-node-pool namespace: strimzi-kafka labels: strimzi.io/cluster: strimzi-kafka-cluster spec: replicas: 1 roles: - controller storage: type: jbod volumes: - id: 0 type: persistent-claim size: 1Gi deleteClaim: true kraftMetadata: shared --- apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaNodePool metadata: name: strimzi-kafka-broker-node-pool namespace: strimzi-kafka labels: strimzi.io/cluster: strimzi-kafka-cluster spec: replicas: 1 roles: - broker storage: type: jbod volumes: - id: 0 type: persistent-claim size: 1Gi deleteClaim: true kraftMetadata: shared --- apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: strimzi-kafka-cluster namespace: strimzi-kafka annotations: strimzi.io/node-pools: enabled strimzi.io/kraft: enabled spec: kafka: version: 3.9.0 metadataVersion: "3.9" listeners: - name: plain port: 9092 type: internal tls: false - name: tls port: 9093 type: internal tls: true - name: external port: 9094 type: ingress tls: true authentication: type: tls configuration: bootstrap: host: strimzi-kafka.172.18.0.3.sslip.io brokerCertChainAndKey: certificate: tls.crt key: tls.key secretName: strimzi-kafka-cluster-tls brokers: - broker: 0 host: strimzi-kafka-broker-0.172.18.0.3.sslip.io class: nginx config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 entityOperator: topicOperator: {} userOperator: {}
Remember to replace the DNS hostnames if your ingress controller LoadBalancer IP was different.

Apply the YAML manifests:
kubectl apply -f strimzi-kafka-1c1b.yaml
Wait for the strimzi-kafka-cluster Kafka cluster to become ready.
kubectl -n strimzi-kafka wait \ --for=condition=Ready \ --timeout=300s \ kafkas.kafka.strimzi.io \ strimzi-kafka-cluster
Sample output:
kafka.kafka.strimzi.io/strimzi-kafka-cluster condition met
The Kafka controller, broker and entity operator pods are running:
kubectl -n strimzi-kafka get pods
Sample output:
NAME READY STATUS RESTARTS AGE strimzi-kafka-cluster-entity-operator-6b4bfc8c98-769fm 2/2 Running 0 11m strimzi-kafka-cluster-strimzi-kafka-broker-node-pool-0 1/1 Running 0 12m strimzi-kafka-cluster-strimzi-kafka-controller-node-pool-1 1/1 Running 0 12m
Observe that Ingress resources are created as well as per our Kafka configuration. This allows external clients to connect to our Strimzi Kafka cluster over TLS.
kubectl -n strimzi-kafka get ingress
Sample output:
NAME CLASS HOSTS ADDRESS PORTS AGE strimzi-kafka-cluster-kafka-bootstrap nginx strimzi-kafka.172.18.0.3.sslip.io 172.18.0.3 80, 443 15m strimzi-kafka-cluster-strimzi-kafka-broker-node-pool-0 nginx strimzi-kafka-broker-0.172.18.0.3.sslip.io 172.18.0.3 80, 443 15m
Here are a few open-ended exercises for the interested reader:
1. How to connect to the Strimzi Kafka cluster securely from an external client via TLS? Is any additional configuration required?
2. Try changing the authentication type from tls to scram-sha-512, oauth or custom. What other configuration changes are required for each authentication type? Can you make it work?
3. Check out the topic operator, create and experiment with topics created via the KafkaTopic custom resource. What advantages does KafkaTopic bring over manually creating and managing topics from an external client?
4. Check out some of the other custom resources provided by Strimzi as well. For example, KafkaMirrorMaker2
5. Modify the KafkaNodePool and Kafka manifests above to deploy a highly-available Strimzi Kafka cluster with 3 controllers and 3 brokers to a separate namespace strimzi-kafka-ha. What changes are required for each custom resource, if any? Compare the changes to the Strimzi custom resources versus the compose.yaml configuration file required to convert our Kafka clusters to an HA setup
6. Try an in-place scaling operation on our existing Strimzi Kafka cluster from 1 controller, 1 broker to 3 controllers, 3 brokers. Ensure that downtime and data loss is minimized. Is the scaling operation successful? How difficult (easy) is it with Strimzi? Is the entire process automated? How much (if any) manual intervention is required? What if we try the same in-place scaling operation on our Kafka cluster with Docker Compose?
Once you’re done experimenting, simply delete the cluster with a single command:
kind delete cluster
Concluding remarks and going further

Kafka is an open source event streaming platform optimized for real-time data processing. While it supports many deployment options, we saw in this lab how the Strimzi operator enables us to deploy and manage Kafka in an automated fashion much like a managed service but with the added benefit of being vendor-neutral and retaining control over your data. Furthermore, the abstractions provided by Strimzi in the form of custom resources simplify non-trivial configuration options such as TLS and OAuth authentication for Kafka administrators.

I hope you enjoyed this article and stay tuned for more updates 😉
+

+

+

+

+

+

+ Data Streaming, Kafka, Kubernetes

+

+

+

+

+

+

+

+

+

+

+

+
Enforcing recommended practices on Kubernetes with ValidatingAdmissionPolicy

2024-12-28

+

+

+

+

+

+
Recall that each Kubernetes request goes through 3 stages as it is evaluated by the API server:
1. Authentication: who you are
2. Authorization: what actions are allowed
3. Admission control: additional validation and mutation checks before the finalized request is accepted (rejected)
See also: Controlling Access to the Kubernetes API | Kubernetes

While many built-in admission controllers are enabled by default on Kubernetes, it was already possible since the early days of Kubernetes to extend admission control by writing your own admission webhooks. However, a drawback to this approach is that it requires some degree of programming knowledge which not all Kubernetes administrators may readily possess.

To simplify the process of implementing custom admission control logic, ValidatingAdmissionPolicy was introduced as an alpha feature in Kubernetes 1.26 and promoted to GA in 1.30. It allows administrators to easily and quickly define request validation logic with Common Expression Language (CEL) instead of writing code. This lowers the barrier for administrators to enforce compliance controls and recommended practices as well as mitigate platform vulnerabilities in Kubernetes environments.

This lab exercise demonstrates the use of ValidatingAdmissionPolicies to perform the following tasks:
1. Disallow pods in the default namespace as per CIS Kubernetes 5.7.4
2. Mitigate CVE-2024-10220 by rejecting pods with gitRepo volumes
Lab: Enforce Kubernetes compliance controls and mitigate vulnerabilities with ValidatingAdmissionPolicy

This lab has been tested with Kubernetes v1.32 (Penelope).

Prerequisites

Familiarity with Kubernetes is assumed. See LFS158x: Introduction to Kubernetes for a gentle introduction to Kubernetes.

Setting up your environment

A Linux environment with at least 2 vCPUs, 8GiB memory and sufficient available disk space capable of running Docker. This can be your own desktop/laptop if you’re a Linux user (like I am ;-), or a spare board (e.g. Raspberry Pi), physical server, virtual machine or cloud instance.

The reference environment is Ubuntu 24.04 LTS (Noble Numbat) so if you’re on a different Linux distribution, adapt apt-related commands with dnf / pacman / something else accordingly when installing system packages. Otherwise, the remaining instructions should be broadly applicable to most Linux distributions.

Install Docker

We’ll use Docker to spin up a kind Kubernetes cluster. It’s convenient, fast, simple and sufficient for this lab exercise.

Install the Docker engine and add the current user to the docker group:
sudo apt update && sudo apt install -y docker.io sudo usermod -aG docker "${USER}"
Log out and in for the changes to take effect.

Check that we have the correct version of Docker installed:
docker version
Sample output:
Client: Version: 26.1.3 API version: 1.45 Go version: go1.22.2 Git commit: 26.1.3-0ubuntu1~24.04.1 Built: Mon Oct 14 14:29:26 2024 OS/Arch: linux/amd64 Context: default Server: Engine: Version: 26.1.3 API version: 1.45 (minimum version 1.24) Go version: go1.22.2 Git commit: 26.1.3-0ubuntu1~24.04.1 Built: Mon Oct 14 14:29:26 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.12 GitCommit: runc: Version: 1.1.12-0ubuntu3.1 GitCommit: docker-init: Version: 0.19.0 GitCommit:
Install kind and spin up a cluster

Just follow the instructions in their Quickstart:
# For AMD64 / x86_64 [ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.26.0/kind-linux-amd64 # For ARM64 [ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.26.0/kind-linux-arm64 chmod +x ./kind sudo mv ./kind /usr/local/bin/kind
Check the correct kind version is installed:
kind version
Sample output:
kind v0.26.0 go1.23.4 linux/amd64
Now our Kubernetes cluster is but a single command away:
kind create cluster
Install and configure kubectl

Again, the official instructions will suffice:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/.
Check that kubectl is correctly installed:
kubectl version
Sample output:
Client Version: v1.32.0 Kustomize Version: v5.5.0 Server Version: v1.32.0
For command-line completion, add the following line to your ~/.bashrc:
source <(kubectl completion bash)
Now save the file and run:
source ~/.bashrc
Enforce CIS Kubernetes Benchmarks with ValidatingAdmissionPolicy

The CIS Kubernetes Benchmarks includes the following rule which typically requires a manual check:

5.7.4. The default namespace should not be used

Fortunately, we can enforce this rule automatically as a CEL expression with ValidatingAdmissionPolicy. The trivial CEL expression false rejects all requests unconditionally and we can scope its evaluation to Pods only.
--- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: deny-all-pods spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["pods"] validations: - expression: "false"
Save the policy above as deny-all-pods.yaml and apply it with kubectl apply.
kubectl apply -f deny-all-pods.yaml
Sample output:
validatingadmissionpolicy.admissionregistration.k8s.io/deny-all-pods created
View the created ValidatingAdmissionPolicy:
kubectl get validatingadmissionpolicies
Sample output:
NAME VALIDATIONS PARAMKIND AGE deny-all-pods 1 <unset> 7s
Next, apply the policy to the default namespace with the ValidatingAdmissionPolicyBinding below.
--- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: deny-all-pods-default spec: policyName: deny-all-pods validationActions: [Deny] matchResources: namespaceSelector: matchLabels: kubernetes.io/metadata.name: default
Save it as deny-all-pods-default.yaml and apply it to the cluster.
kubectl apply -f deny-all-pods-default.yaml
Sample output:
validatingadmissionpolicybinding.admissionregistration.k8s.io/deny-all-pods-default created
View the created ValidatingAdmissionPolicyBinding:
kubectl get validatingadmissionpolicybindings
Sample output:
NAME POLICYNAME PARAMREF AGE deny-all-pods-default deny-all-pods <unset> 2m8s
Now confirm that we are unable to create pods in the default namespace. Save the YAML below as busybox.yaml:
--- apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: busybox name: busybox spec: containers: - args: - sleep - infinity image: busybox name: busybox resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {}
Try to create the pod with kubectl create and notice how the request is rejected.
kubectl create -f busybox.yaml
Sample output:
The pods "busybox" is invalid: : ValidatingAdmissionPolicy 'deny-all-pods' with binding 'deny-all-pods-default' denied request: failed expression: false
Let’s confirm that we are able to create pods in other namespaces. Create a busybox namespace:
kubectl create ns busybox
Now save the YAML below as busybox-busybox.yaml which defines the same Busybox pod but in the busybox namespace instead of the default namespace.
--- apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: busybox name: busybox namespace: busybox spec: containers: - args: - sleep - infinity image: busybox name: busybox resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {}
Create the pod with kubectl create and confirm that the pod is created.
kubectl create -f busybox-busybox.yaml
Sample output:
pod/busybox created
Now confirm the pod is running.
kubectl -n busybox get pods
Sample output:
NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 58s
Delete the pod – we’ll re-create it afterwards.
kubectl delete -f busybox-busybox.yaml
Mitigate CVE-2024-10220 with ValidatingAdmissionPolicy

CVE-2024-10220 is a high severity vulnerability with real-world exploits by leveraging gitRepo volumes deprecated since Kubernetes 1.11.

The Kubernetes kubelet component allows arbitrary command execution via specially crafted gitRepo volumes.

While this vulnerability was patched in Kubernetes 1.31 and backported to 1.29.7 and 1.30.3, Kubernetes upstream recommends replacing existing gitRepo volumes with emptyDir volumes initialized via init containers and there is no reason for legitimate workloads to use gitRepo volumes today. Therefore, it would make sense to define a ValidatingAdmissionPolicy and corresponding ValidatingAdmissionPolicyBinding to disallow the use of gitRepo volumes entirely across all namespaces.

The CEL expression for rejecting gitRepo volumes below is taken from the official documentation.
!has(object.spec.volumes) || !object.spec.volumes.exists(v, has(v.gitRepo))
The ValidatingAdmissionPolicy – name it forbid-gitrepo-volumes.yaml:
--- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: forbid-gitrepo-volumes spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["pods"] validations: - expression: "!has(object.spec.volumes) || !object.spec.volumes.exists(v, has(v.gitRepo))"
The associated ValidatingAdmissionPolicyBinding – name it forbid-gitrepo-volumes-binding.yaml:
--- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: forbid-gitrepo-volumes-binding spec: policyName: forbid-gitrepo-volumes validationActions: [Deny]
Create the policy and associated binding.
kubectl create -f forbid-gitrepo-volumes.yaml kubectl create -f forbid-gitrepo-volumes-binding.yaml
Confirm we can create our Busybox pod in the busybox namespace since it does not contain a gitRepo volume:
kubectl create -f busybox-busybox.yaml
Sample output:
pod/busybox created
Now prepare a modified Busybox pod with a gitRepo volume – name the file busybox-gitrepo-busybox.yaml.
--- apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: busybox name: busybox-gitrepo namespace: busybox spec: containers: - args: - sleep - infinity image: busybox name: busybox resources: {} volumeMounts: - name: my-gitrepo-volume mountPath: /srv/source dnsPolicy: ClusterFirst restartPolicy: Always volumes: - name: my-gitrepo-volume gitRepo: repository: git@github.com:kubernetes/kubernetes revision: 7bfdda4696f78fc789fe91420f7c8609c71002d0 status: {}
Try to create the pod with kubectl create and observe the failure.
kubectl create -f busybox-gitrepo-busybox.yaml
Sample output:
Warning: spec.volumes[0].gitRepo: deprecated in v1.11 The pods "busybox-gitrepo" is invalid: : ValidatingAdmissionPolicy 'forbid-gitrepo-volumes' with binding 'forbid-gitrepo-volumes-binding' denied request: failed expression: !has(object.spec.volumes) || !object.spec.volumes.exists(v, has(v.gitRepo))
Concluding remarks and going further

While ValidatingAdmissionPolicy allows Kubernetes administrators to define resource validation logic with CEL expressions without writing code, MutatingAdmissionPolicy was recently introduced in Kubernetes 1.32 as an alpha feature allowing resource mutation logic to be similarly defined as a combination of CEL expressions and JSON patches without writing code. However, as an alpha feature, it is disabled by default and hidden behind a feature gate which must be explicitly enabled.

Kubernetes-native admission policies enable Kubernetes administrators to enforce policies with ease at the cluster, namespace and resource level. While they serve as a ready replacement for admission webhooks and perhaps even some simple use cases of policy engines such as OPA Gatekeeper, third-party policy engines such as OPA Gatekeeper, Kyverno and Kubewarden often provide additional features and guarantees beyond the scope of upstream Kubernetes useful for seasonsed Kubernetes administrators:
- Integrated CLI support
- Graphical user interface
- Advanced policy reporting
- Enterprise support plans
- … etc.
+

+

+

+

+

+

+ Kubernetes, Security

+

+

+

+

+

+

+

+

+

+

+

+
Protecting stateful workloads on Kubernetes with Kanister

2024-06-09

+

+

+

+

+

+
The source code for this lab exercise is available on GitHub.

In our previous article Running stateful workloads on Kubernetes with Rook Ceph, we saw how Kubernetes CSI enables us to take volume snapshots on supported storage backends as a first step towards protecting our data on Kubernetes. However, snapshots operate at the infrastructure level so they do not understand how applications operate, manage and structure their data. This implies that snapshots, by nature, are crash-consistent but not application-consistent. For busy stateful workloads such as databases processing many transactions per second, crash-consistency is insufficient for data protection since in-progress transactions are not recorded so restoring from a snapshot may still lead to data loss and leave the application in an inconsistent state.

Kanister provides a robust and flexible solution for defining your own actions for performing application-aware backups on Kubernetes. It is a CNCF sandbox project originally created by the Veeam Kasten team as an integral component of their enterprise-ready Kubernetes data protection platform. It does this by defining blueprints, which serve as templates for application-specific backup and restore logic. The backup administrator or application owner may then instantiate actions defined in these blueprints by creating ActionSets which perform the actual application-specific backup and recovery procedures.

This lab exercise demonstrates how to back up and restore WordPress on Kubernetes with Kanister in a reliable manner, by creating a logical database backup (database dump) and exporting it to S3 which can be imported during the restore phase to return WordPress to a known good state. The backup procedure consists of the following steps:
1. Scale the WordPress deployment to zero to stop accepting user traffic and complete pending database transactions
2. Take a logical dump of the database and upload it to S3
3. Scale the WordPress deployment back to the original size to start accepting user traffic again
The restore procedure is also similar:
1. Scale the WordPress deployment to zero to stop accepting user traffic and ensure no additional database transactions are made during the restore operation
2. Download the logical database dump from S3 and import it to our running database
3. Scale the WordPress deployment back to the original size to start accepting user traffic again
Lab: Back up and restore WordPress on Kubernetes with Kanister

This lab has been tested with Kubernetes v1.30 (Uwubernetes).

Prerequisites

Familiarity with running stateful workloads on Kubernetes is assumed. Before proceeding with this lab exercise, consider checking out my other articles on Kubernetes storage as well:
1. Investigating a failed VolumeSnapshot with NFS on Kubernetes
2. Running stateful workloads on Kubernetes with Rook Ceph
Setting up your environment

A Linux environment with at least 2 vCPUs, 8GiB memory and sufficient available disk space capable of running Docker. This can be your own desktop/laptop if you’re a Linux user (like I am ;-), or a spare board (e.g. Raspberry Pi), physical server, virtual machine or cloud instance. You’ll also need an AWS account so sign up for one if you haven’t already – the Free Tier is quite promising for new users to AWS. Otherwise, you can use any S3 compatible object storage like MinIO but beware that you’ll have to create the bucket manually and adapt some of the remaining instructions accordingly.

The reference environment is Ubuntu 24.04 LTS (Noble Numbat) so if you’re on a different Linux distribution, adapt apt-related commands with dnf / pacman / something else accordingly when installing system packages. Otherwise, the remaining instructions should be broadly applicable to most Linux distributions.

Configure your AWS credentials

Create an IAM administrator account, then generate access and secret keys for that account and configure your AWS credentials for programmatic access. The simplest way to do so and confirm that you have your AWS credentials set up correctly is by installing and setting up AWS CLI v2, then running a simple command such as the one below as a sanity check:
aws ec2 describe-instances
Sample output:
{ "Reservations": [] }
Install OpenTofu

We’ll use OpenTofu to create our S3 bucket and generate restricted IAM credentials automatically for performing backup and restore operations to and from S3. OpenTofu is an open-source fork of Terraform compatible with legacy Terraform (<= 1.5).

The latest version at the time of writing is 1.7.2.
wget https://github.com/opentofu/opentofu/releases/download/v1.7.2/tofu_1.7.2_linux_amd64.tar.gz tar xvf tofu_1.7.2_linux_amd64.tar.gz chmod +x ./tofu sudo mv ./tofu /usr/local/bin/.
Check that we have the correct version installed:
tofu version
Sample output:
OpenTofu v1.7.2 on linux_amd64
Install Docker

We’ll use Docker to spin up a kind Kubernetes cluster. It’s convenient, fast, simple and sufficient for this lab exercise.

Install the Docker engine and add the current user to the docker group:
sudo apt update && sudo apt install -y docker.io sudo usermod -aG docker "${USER}"
Log out and in for the changes to take effect.

Check that we have the correct version of Docker installed:
docker version
Sample output:
Client: Version: 24.0.7 API version: 1.43 Go version: go1.22.2 Git commit: 24.0.7-0ubuntu4 Built: Wed Apr 17 20:08:25 2024 OS/Arch: linux/amd64 Context: default Server: Engine: Version: 24.0.7 API version: 1.43 (minimum version 1.12) Go version: go1.22.2 Git commit: 24.0.7-0ubuntu4 Built: Wed Apr 17 20:08:25 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.12 GitCommit: runc: Version: 1.1.12-0ubuntu3 GitCommit: docker-init: Version: 0.19.0 GitCommit:
Install kind and spin up a cluster

Just follow the instructions in their Quickstart:
# For AMD64 / x86_64 [ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-linux-amd64 # For ARM64 [ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-linux-arm64 chmod +x ./kind sudo mv ./kind /usr/local/bin/kind
Check the correct kind version is installed:
kind version
Sample output:
kind v0.23.0 go1.21.10 linux/amd64
Now our Kubernetes cluster is but a single command away:
kind create cluster
Install and configure kubectl

Again, the official instructions will suffice:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/.
Check that kubectl is correctly installed:
kubectl version
Sample output:
Client Version: v1.30.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.0
For command-line completion, add the following line to your ~/.bashrc:
source <(kubectl completion bash)
Now save the file and run:
source ~/.bashrc
Install and configure Helm

The latest version is 3.15.1 at the time of writing.
wget https://get.helm.sh/helm-v3.15.1-linux-amd64.tar.gz tar xvf helm-v3.15.1-linux-amd64.tar.gz chmod +x linux-amd64/helm sudo mv linux-amd64/helm /usr/local/bin/.
Check that Helm is installed:
helm version
Sample output:
version.BuildInfo{Version:"v3.15.1", GitCommit:"e211f2aa62992bd72586b395de50979e31231829", GitTreeState:"clean", GoVersion:"go1.22.3"}
Optionally append the following line to your ~/.bashrc and source it for Helm command-line completion:
source <(helm completion bash)
Install Go 1.22

Go is the programming language underpinning Kubernetes and much of the cloud native ecosystem. Not surprisingly, Kanister is written in Go as well.

We need to install the Go SDK and toolchain for building and installing the Kanister command-line tools from source. Unfortunately, there seems to be no officially published binaries that can be downloaded directly.
wget https://go.dev/dl/go1.22.4.linux-amd64.tar.gz sudo bash -c "rm -rf /usr/local/go && tar -C /usr/local -xzf go1.22.4.linux-amd64.tar.gz"
Now append the following line to your ~/.profile:
export PATH="$PATH:/usr/local/go/bin"
Log out and in again for the changes to take effect.

Confirm the correct version of Go is installed:
go version
Sample output:
go version go1.22.4 linux/amd64
Install the Kanister tools

Follow the official instructions here as well:
curl https://raw.githubusercontent.com/kanisterio/kanister/master/scripts/get.sh | bash
Check that kanctl is installed which we’ll use later:
kanctl --version
Sample output:
kanctl version {"version": "0.109.0", "gitCommit": "568148b76a38064d716025c0b639eb398f2dc782", "buildDate": "2024-05-23T02:33:07Z"}
Installing WordPress

With all that out of the way, we’re now all set to install WordPress on our cluster.

Let’s use the Helm chart published by Bitnami. Add the Bitnami repo and refresh repository metadata:
helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update
Now install WordPress in a new namespace wordpress with the default options and the release name wordpress:
helm -n wordpress install \ wordpress \ bitnami/wordpress \ --version 22.4.8 \ --create-namespace
Sample output:
NAME: wordpress LAST DEPLOYED: Sun Jun 9 08:48:13 2024 NAMESPACE: wordpress STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: wordpress CHART VERSION: 22.4.8 APP VERSION: 6.5.4 ...
Wait for all pods in our WordPress instance to become ready:
kubectl -n wordpress wait \ --for=condition=Ready \ pods \ --all \ --timeout=180s
Sample output:
pod/wordpress-5cffb559cf-wqp2k condition met pod/wordpress-mariadb-0 condition met
Installing the Kanister operator

The Kanister operator is responsible for managing Kanister-specific custom resources such as blueprints, ActionSets and location profiles. Install the operator via the official Helm chart.

Add the Kanister repository and refresh repository metadata:
helm repo add kanister https://charts.kanister.io/ helm repo update
Now install it in the kanister namespace with the default options:
helm -n kanister install \ kanister \ kanister/kanister-operator \ --create-namespace
Wait for the operator to become ready:
kubectl -n kanister wait \ --for=condition=Ready \ pods \ -l app=kanister-operator \ --timeout=180s
Sample output:
pod/kanister-kanister-operator-549c65f8c9-r29vj condition met
Creating our S3 bucket for storing our Kanister backups

Clone the repository for this lab exercise and navigate to the project directory:
git clone https://github.com/DonaldKellett/kanister-wordpress.git cd kanister-wordpress/
Now initialize OpenTofu and apply the configuration. Answer yes when prompted:
tofu init tofu apply
The S3 bucket is now created and the manifests for the location profile with the corresponding secret generated under manifests/ for pointing Kanister to our S3 bucket during the backup and restore operations:
- manifests/secret.yaml
- manifests/profile.yaml
Feel free to check out the blueprint as well:
- manifests/blueprint.yaml
The blueprint is responsible for defining the following actions:
- quiesce: Scale the WordPress deployment to zero before a backup / restore operation
- unquiesce: the opposite of quiesce which scales the WordPress deployment back up to its original size
- backup: Performs a logical dump of the WordPress database and uploads it to S3
- restore: Fetches a remote database dump from S3 and imports it to our running database
Backing up our WordPress database to S3

We’re all set to back up our WordPress database to S3, but before that, let’s take a look at our WordPress instance.

Port-forward the wordpress service in the wordpress namespace to port 8080:
kubectl -n wordpress port-forward svc/wordpress 8080:80
Leave the current terminal window open and open a new window (tab) to run subsequent commands in this lab. Now open your browser and visit the page at http://localhost:8080/. Notice that there is a single “Hello World” blog post.

Return to the command line and create the blueprint:
kubectl create -f manifests/blueprint.yaml
Create a location profile and corresponding secret as well to point to our S3 bucket where we’ll store our database dumps to.
kubectl create -f manifests/secret.yaml kubectl create -f manifests/profile.yaml
Now use kanctl to run the quiesce action in our blueprint. This causes WordPress to drop all user traffic so pending database transactions are allowed to complete and no new database transactions are initiated by the frontend.
kanctl -n kanister create actionset \ --action quiesce \ --blueprint wordpress-bp \ --deployment wordpress/wordpress
Make note of the name of the created ActionSet since we’ll need to refer to it later during the unquiesce operation:
actionset quiesce-pp78r created
Wait for the ActionSet to complete – replace the variable QUIESCE_ACTIONSET with your ActionSet name above:
QUIESCE_ACTIONSET="quiesce-pp78r" # Replace me! kubectl -n kanister wait \ --for=jsonpath='{.status.state}'=complete \ actionsets.cr.kanister.io \ "${QUIESCE_ACTIONSET}" \ --timeout=180s
Sample output:
actionset.cr.kanister.io/quiesce-pp78r condition met
Now run the backup action – again, make note of the name since we’ll need it for the restore process:
kanctl -n kanister create actionset \ --action backup \ --blueprint wordpress-bp \ --profile wordpress-s3-profile \ --statefulset wordpress/wordpress-mariadb
Sample output:
actionset backup-sm8pn created
Wait for the backup ActionSet to complete – once again, replace the variable as appropriate:
BACKUP_ACTIONSET="backup-sm8pn" # Replace me! kubectl -n kanister wait \ --for=jsonpath='{.status.state}'=complete \ actionsets.cr.kanister.io \ "${BACKUP_ACTIONSET}" \ --timeout=180s
Sample output:
actionset.cr.kanister.io/backup-sm8pn condition met
Now un-quiesce our WordPress application so it can serve user requests again:
QUIESCE_ACTIONSET="quiesce-pp78r" # Replace me! kanctl -n kanister create actionset \ --action unquiesce \ --from "${QUIESCE_ACTIONSET}"
Sample output:
actionset unquiesce-quiesce-pp78r-kdkgv created
Wait once again for the operation to complete:
UNQUIESCE_ACTIONSET="unquiesce-quiesce-pp78r-kdkgv" # Replace me! kubectl -n kanister wait \ --for=jsonpath='{.status.state}'=complete \ actionsets.cr.kanister.io \ "${UNQUIESCE_ACTIONSET}" \ --timeout=180s
Sample output:
actionset.cr.kanister.io/unquiesce-quiesce-pp78r-kdkgv condition met
“Accidentally” deleting our Hello World blog post

At this point, our port-forward command lost connection to the previous pod due to the quiesce operation so establish the connection again:
kubectl -n wordpress port-forward svc/wordpress 8080:80
Now log in to the WordPress administrator dashboard by pointing your browser to http://localhost:8080/wp-admin/ and entering the following credentials:
- Username: user
- Password: fetch the password from the wordpress-password key of the wordpress secret
To fetch the password, run the following command:
kubectl -n wordpress get secret \ wordpress \ -o jsonpath='{.data.wordpress-password}' | \ base64 -d -
Once logged in to the dashboard, click “At a Glance > 1 Post” to view the published blog posts, proceed to move the Hello World blog post to the trash and confirm the deletion by deleting it permanently.

Return to the admin dashboard and confirm that no blog posts are remaining. Oops – we’ve accidentally deleted our very important blog post!

Restoring our WordPress database from S3

Fortunately, we backed our database to S3 so we can restore our WordPress instance to a known good state.

Quiesce our WordPress application again and take note of the ActionSet name:
kanctl -n kanister create actionset \ --action quiesce \ --blueprint wordpress-bp \ --deployment wordpress/wordpress
Sample output:
actionset quiesce-ms6z6 created
Wait for the quiesce operation to complete:
QUIESCE_ACTIONSET="quiesce-ms6z6" # Replace me! kubectl -n kanister wait \ --for=jsonpath='{.status.state}'=complete \ actionsets.cr.kanister.io \ "${QUIESCE_ACTIONSET}" \ --timeout=180s
Sample output:
actionset.cr.kanister.io/quiesce-ms6z6 condition met
Now run the restore action from the backup we just created earlier:
BACKUP_ACTIONSET="backup-sm8pn" # Replace me! kanctl -n kanister create actionset \ --action restore \ --from "${BACKUP_ACTIONSET}"
Sample output:
actionset restore-backup-sm8pn-lwvck created
Wait for the restore action to complete:
RESTORE_ACTIONSET="restore-backup-sm8pn-lwvck" # Replace me! kubectl -n kanister wait \ --for=jsonpath='{.status.state}'=complete \ actionsets.cr.kanister.io \ "${RESTORE_ACTIONSET}" \ --timeout=180s
Sample output:
actionset.cr.kanister.io/restore-backup-sm8pn-lwvck condition met
Unquiesce our WordPress instance once more:
QUIESCE_ACTIONSET="quiesce-ms6z6" # Replace me! kanctl -n kanister create actionset \ --action unquiesce \ --from "${QUIESCE_ACTIONSET}"
Sample output:
actionset unquiesce-quiesce-ms6z6-z74s8 created
Wait for the unquiesce action to complete:
UNQUIESCE_ACTIONSET="unquiesce-quiesce-ms6z6-z74s8" # Replace me! kubectl -n kanister wait \ --for=jsonpath='{.status.state}'=complete \ actionsets.cr.kanister.io \ "${UNQUIESCE_ACTIONSET}" \ --timeout=180s
Sample output:
actionset.cr.kanister.io/unquiesce-quiesce-ms6z6-z74s8 condition met
Re-establish the connection for the port-forward command which was lost again due to the quiesce operation:
kubectl -n wordpress port-forward svc/wordpress 8080:80
Observe that the Hello World blog post is successfully restored:

Congratulations! You successfully backed up and restored WordPress on Kubernetes with Kanister!

Concluding remarks and going further

We saw how Kanister can be used to define your own blueprints and actions to perform application-specific backup and recovery operations on Kubernetes. This ensures that your backups are application-consistent and can be safely restored from in case of human error or partial storage failure without compromsing on the consistency of the data from the application’s perspective.

For a comprehensive enterprise-ready Kubernetes backup and disaster recovery (DR) solution suitable for production Kubernetes environments, do check out Veeam Kasten (formerly Kasten K10) as well which is available for evaluation at no cost for small non-production clusters with up to 5 nodes.
+

+

+

+

+

+

+ Data Protection and Governance, Kubernetes, Storage, WordPress

+

+

+

+

+

+

+

+

+

+

+

+

Deploying Keycloak on Fedora Workstation with Podman

2024-02-25

Keycloak is an open source identity provider (IdP) with single-sign on (SSO) capabilities. It supports the most widely used enterprise authentication protocols, namely OpenID Connect (OIDC), OAuth 2.0, and SAML. With Keycloak, users sign in once and share the same identity across multiple applications and platforms in a transparent manner. On the other hand, applications defer authentication to Keycloak using a standard protocol without managing the authentication flow themselves.

Keycloak boasts several advantages over competing solutions such as Dex and Authentik:

Officially governed by the Cloud Native Computing Foundation (CNCF) as an Incubating project
Offers an user-friendly web interface with extensive features for simplified administration
Supports multi-tenancy out of the box via the concept of realms
Offers direct integration with a multitude of IdP backends such as Active Directory and GitHub
Supports the enforcement of multi-factor authentication (MFA) for both individual users and realms

This article will guide you through setting up Keycloak on Fedora Linux with SSL/TLS enabled using Podman. Once our Keycloak server is up and running, we will:

Create our first realm example-com representing our organization
Enforce mandatory MFA for all users in our realm
Create our first user johndoe representing an employee in our organization
Create our first client keycloak-spa corresponding to the Keycloak demo single-page application (SPA)
Log in to our Keycloak realm via the demo application to validate our configuration

System requirements

A physical or virtual machine running Fedora Workstation with the following specifications:

Minimum 4G of RAM (8G+ recommended)
Minimum 2 vCPU cores (4+ vCPU recommended)
Sufficient available storage – a few GB should suffice

Generating certificates for SSL/TLS

Applications deferring authentication to Keycloak exchange sensitive credentials during the authentication process. As such, never run Keycloak with plain HTTP in production as it transmits credentials in plaintext over the network. Instead, we will generate the necessary SSL/TLS certificates, add our custom root certificate authority (CA) to the system trust and start Keycloak with HTTPs using our certificates to ensure a secure, encrypted connection.

Install the OpenSSL package used for generating SSL/TLS certificates:

sudo dnf install --refresh -y openssl

Next, create the directory $HOME/ssl/ for our certificates and enter it:

			
mkdir -p "$HOME/ssl/"
pushd "$HOME/ssl/"

Now generate a CA key and certificate for internal use within our organization example.com. We will use the CA key to sign certificate signing requests (CSRs) resulting in SSL/TLS certificates used to encrypt HTTPs traffic for internal websites and services, and use the CA certificate to verify the validity of the SSL/TLS certificates presented by internal websites and services. CAs serve as an anchor of trust so their certificates should be long lived, e.g. with a validity period of 10 years.

			
openssl req \
    -x509 \
    -new \
    -nodes \
    -keyout my-custom-ca.key \
    -sha256 \
    -days 3650 \
    -out my-custom-ca.crt \
    -subj '/CN=My custom CA/DC=localhost'

		

This creates a CA key my-custom-ca.key and the corresponding CA certificate my-custom-ca.crt.

Inspect the CA certificate with the following command:

openssl x509 -text -noout -in my-custom-ca.crt

Sample output:

			
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            76:56:b9:13:b4:85:f4:36:0d:d0:2a:18:d3:de:f3:2f:25:76:f5:b0
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = My custom CA, DC = localhost
        Validity
            Not Before: Feb 25 02:11:46 2024 GMT
            Not After : Feb 22 02:11:46 2034 GMT
        Subject: CN = My custom CA, DC = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ba:0b:7e:fb:88:e6:6f:f3:c2:3b:cc:3c:5d:18:
                    cf:bc:c1:ff:e3:98:3a:52:2c:82:dc:d7:14:75:8a:
                    58:fc:26:d7:97:9e:ee:73:c0:56:79:77:5b:97:96:
                    5c:f9:6a:d1:0c:bc:96:4d:e3:48:4c:e8:91:77:1e:
                    58:22:23:b2:7d:50:2e:2e:5d:c5:7a:85:0e:e4:b4:
                    73:64:5d:85:2a:81:fe:e1:04:3d:6a:5d:0f:99:4b:
                    e8:82:7b:c4:06:36:3f:67:b7:19:08:1b:56:28:d0:
                    b1:0d:f5:52:55:06:88:97:87:a5:be:ba:11:a2:eb:
                    a4:da:6d:c7:3e:1f:5d:46:1f:da:a3:26:19:0a:ea:
                    42:67:8e:ab:7a:77:d5:db:c0:24:d1:c7:10:86:b9:
                    ef:0a:ad:3f:22:8d:17:f5:26:1e:38:23:ba:5f:7c:
                    6b:f6:e8:41:1f:ed:6a:a6:77:3e:f6:55:ef:06:08:
                    f3:a8:f3:a3:4f:9b:d4:2c:fe:b0:62:c1:1d:8e:c0:
                    6f:97:d9:e4:ec:a7:e6:6d:62:31:8a:f7:3e:9a:4f:
                    3b:92:c4:04:b6:2c:9b:b2:4d:b8:bc:8a:72:c1:67:
                    9f:8a:60:cf:a5:7f:84:75:a3:c9:48:60:dd:6a:b9:
                    e4:ec:91:3f:6d:cc:ec:a3:8a:44:ea:1c:62:01:92:
                    09:2d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                69:0F:42:C2:FC:D7:BB:79:D3:B6:5A:31:D9:A9:B0:3C:DA:C3:78:1D
            X509v3 Authority Key Identifier: 
                69:0F:42:C2:FC:D7:BB:79:D3:B6:5A:31:D9:A9:B0:3C:DA:C3:78:1D
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        8d:15:51:ae:d6:67:6f:18:b2:88:30:c2:91:a8:25:16:3c:97:
        19:f1:c6:d4:4a:95:0f:ec:21:97:f8:aa:ec:78:5d:22:66:4e:
        84:a6:e3:61:be:bd:a2:b3:6a:48:f7:0e:95:f5:95:02:14:63:
        28:cc:c8:6d:07:86:3d:9d:f1:f5:8b:5c:a9:4f:d2:98:0d:1a:
        26:cd:3e:ef:8c:e7:0c:e6:10:31:94:ad:c3:6c:83:82:d9:7e:
        fe:c7:85:c4:40:d7:36:7a:fb:6e:79:f9:c4:64:77:59:d6:6c:
        f5:24:36:fd:f4:2a:97:ec:58:0c:c2:6c:16:87:d6:4f:7e:88:
        9d:16:c8:fa:5f:aa:44:52:3b:36:99:34:64:e2:b9:a8:1b:e8:
        f2:94:95:ab:ae:12:42:ab:11:4d:9b:e5:c3:5c:85:07:2c:1e:
        7e:2e:ef:88:ad:44:eb:90:a0:b1:28:c5:a6:4f:0e:1c:47:6a:
        34:da:a0:5a:1a:9a:a9:b0:fe:77:f1:e3:99:90:59:47:4b:52:
        d2:e5:5d:9b:46:6e:04:fe:4f:87:75:13:ab:0a:a8:74:c4:c8:
        80:54:cd:6b:db:4a:ee:86:bc:5c:32:9e:5f:29:8b:50:3f:0c:
        ab:32:49:ff:ce:36:9c:cd:e4:4c:78:da:ce:a1:4d:a5:a3:84:
        51:41:52:49

		

Look out for the following fields when inspecting a CA certificate:

Issuer: the subject of the authority who issued the certificate. In the case of a root CA, this is the CA itself
Subject: who this certificate represents
X509v3 extensions: metadata indicating the nature of the certificate (CA:TRUE) and associated identity information

Be accustomed to inspecting certificates with the command shown above as it is useful for troubleshooting when certificates do not function as expected, e.g. when your browser warns that a certificate is invalid even though you already trust the root CA that signed it.

Before we proceed, add the CA certificate to our system trust. This ensures that any certificates issued by this CA are recognized as valid and trustworthy.

			
sudo trust anchor --store my-custom-ca.crt
sudo update-ca-trust

Now generate a CSR for our Keycloak instance which will be exposed on https://localhost:8443/.

			
openssl req \
    -new \
    -newkey rsa:2048 \
    -nodes \
    -keyout localhost.key \
    -sha256 \
    -out localhost.csr \
    -subj '/CN=My custom certificate/DC=localhost' \
    -addext 'subjectAltName=DNS:localhost' \
    -addext 'basicConstraints=critical,CA:FALSE' \
    -addext 'keyUsage=digitalSignature' \
    -addext 'extendedKeyUsage=serverAuth'

		

This generates a certificate key localhost.key and the corresponding CSR localhost.csr. The CSR is then handed over to the CA for signing which generates an SSL/TLS certificate for use in our Keycloak instance.

Inspect the CSR with a similar command:

openssl req -text -noout -in localhost.csr

Sample output:

			
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = My custom certificate, DC = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:dc:15:3a:92:03:aa:26:c9:1f:b2:a7:fb:3a:70:
                    6a:64:cc:f7:2e:68:88:3b:44:88:d7:f7:59:86:4c:
                    94:c5:88:65:4d:4c:2c:f5:e3:6d:02:8b:e9:16:b8:
                    fc:b1:7b:e1:ca:dd:17:12:75:7e:3b:97:78:01:bd:
                    e9:e4:a2:9c:c3:62:74:e6:32:00:6a:3f:9a:4d:e6:
                    d5:46:b0:c3:79:40:d3:8c:18:6c:41:0b:b8:6f:84:
                    15:bf:37:29:7f:46:d9:5d:0f:2a:d9:06:21:1d:17:
                    64:fb:ec:9f:c1:62:0e:e5:df:48:cf:c9:0e:ea:3e:
                    47:4e:dd:00:95:ac:38:fb:03:bf:8b:13:34:57:49:
                    01:26:05:06:d5:dc:39:34:12:2e:2b:f5:a7:3a:d1:
                    03:94:ce:5c:2b:28:67:2e:2d:09:85:b6:53:0a:f3:
                    9b:00:62:29:d5:84:df:e0:b4:a9:0a:81:a1:f3:0a:
                    e3:fc:71:5d:fe:d7:c7:8b:f7:e5:bb:e6:51:0a:5f:
                    d5:06:5d:7f:cf:48:28:96:b6:d0:a4:60:6f:f5:08:
                    0f:ce:6f:a3:8f:c4:b7:db:c0:79:33:1b:02:e8:ad:
                    95:fc:df:b9:6a:ef:31:f2:1c:11:3c:56:4a:4c:07:
                    1e:53:24:d6:36:2a:e3:3e:4f:3b:b8:64:68:56:70:
                    81:29
                Exponent: 65537 (0x10001)
        Attributes:
            Requested Extensions:
                X509v3 Subject Alternative Name: 
                    DNS:localhost
                X509v3 Basic Constraints: critical
                    CA:FALSE
                X509v3 Key Usage: 
                    Digital Signature
                X509v3 Extended Key Usage: 
                    TLS Web Server Authentication
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        ce:e7:20:cd:f4:86:86:0e:66:95:4d:59:1c:8b:94:9e:8b:dd:
        97:ac:96:8e:bd:ea:96:c7:e4:2f:a8:af:fa:71:e9:aa:ce:85:
        f1:bd:62:a5:8f:5b:07:f1:25:f0:c9:0f:b1:9e:ef:85:af:fc:
        cd:1b:09:e0:c5:41:0b:26:67:35:32:9e:8a:c8:b5:97:14:d6:
        74:e6:dc:9b:de:bf:06:b1:10:28:29:33:c0:48:7c:93:28:29:
        1b:92:43:ae:0c:2c:37:d0:ba:ac:bc:92:c9:61:e8:d1:ba:67:
        ab:36:ee:cb:ad:bc:df:7e:12:92:e4:95:34:a9:4c:c0:8d:3b:
        90:01:37:b0:bf:9b:41:94:52:a6:a9:a2:9b:87:f8:11:bf:80:
        4e:68:6a:37:11:82:12:50:56:f1:e8:fb:2c:1e:5d:e2:7f:27:
        f9:18:f0:32:e0:d8:75:e6:e2:be:4d:62:10:5d:ac:cb:fd:22:
        65:3d:84:74:cf:f6:34:78:ea:85:29:49:52:b2:5d:96:ff:40:
        cb:e7:cb:4d:0a:c3:da:8b:af:92:5d:51:74:98:49:7a:66:da:
        33:ef:5e:95:ca:f0:7a:2b:19:bd:82:56:2f:c1:08:8e:33:37:
        7a:76:88:cb:81:9c:56:77:dc:ac:6c:b4:a0:c2:8d:ff:8e:6e:
        a6:7e:dd:4f

		

Here, pay attention to the requested extensions, in particular the X509v3 Subject Alternative Name which defines the DNS domain name(s) for which the signed certificate will be valid for.

Pass this CSR over to our CA for signing. SSL/TLS certificates should be renewed on a regular basis so set the validity period to a shorter interval, e.g. 1 year.

			
openssl x509 \
    -req \
    -days 365 \
    -in localhost.csr \
    -CA my-custom-ca.crt \
    -CAkey my-custom-ca.key \
    -out localhost.crt \
    -set_serial 01 \
    -sha256 \
    -copy_extensions copy

		

The -copy-extensions copy option preserves the CSR requested extensions in the signed certificate localhost.crt. Confirm this is the case:

openssl x509 -text -noout -in localhost.crt

Sample output:

			
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = My custom CA, DC = localhost
        Validity
            Not Before: Feb 25 03:31:09 2024 GMT
            Not After : Feb 24 03:31:09 2025 GMT
        Subject: CN = My custom certificate, DC = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:dc:15:3a:92:03:aa:26:c9:1f:b2:a7:fb:3a:70:
                    6a:64:cc:f7:2e:68:88:3b:44:88:d7:f7:59:86:4c:
                    94:c5:88:65:4d:4c:2c:f5:e3:6d:02:8b:e9:16:b8:
                    fc:b1:7b:e1:ca:dd:17:12:75:7e:3b:97:78:01:bd:
                    e9:e4:a2:9c:c3:62:74:e6:32:00:6a:3f:9a:4d:e6:
                    d5:46:b0:c3:79:40:d3:8c:18:6c:41:0b:b8:6f:84:
                    15:bf:37:29:7f:46:d9:5d:0f:2a:d9:06:21:1d:17:
                    64:fb:ec:9f:c1:62:0e:e5:df:48:cf:c9:0e:ea:3e:
                    47:4e:dd:00:95:ac:38:fb:03:bf:8b:13:34:57:49:
                    01:26:05:06:d5:dc:39:34:12:2e:2b:f5:a7:3a:d1:
                    03:94:ce:5c:2b:28:67:2e:2d:09:85:b6:53:0a:f3:
                    9b:00:62:29:d5:84:df:e0:b4:a9:0a:81:a1:f3:0a:
                    e3:fc:71:5d:fe:d7:c7:8b:f7:e5:bb:e6:51:0a:5f:
                    d5:06:5d:7f:cf:48:28:96:b6:d0:a4:60:6f:f5:08:
                    0f:ce:6f:a3:8f:c4:b7:db:c0:79:33:1b:02:e8:ad:
                    95:fc:df:b9:6a:ef:31:f2:1c:11:3c:56:4a:4c:07:
                    1e:53:24:d6:36:2a:e3:3e:4f:3b:b8:64:68:56:70:
                    81:29
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:localhost
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Subject Key Identifier: 
                F0:5D:A9:79:38:3B:83:EA:D0:C6:CF:91:0E:6F:67:72:DE:45:3A:04
            X509v3 Authority Key Identifier: 
                A1:EA:2F:AB:B1:95:AE:9C:56:48:1C:B5:6B:42:8A:61:2C:0B:DB:18
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        c1:b0:c6:89:64:76:b1:ab:2e:bc:c7:fd:bd:d2:31:9f:20:8d:
        c0:8e:1d:d7:70:f1:a6:72:7d:49:23:33:f3:f5:e6:2d:eb:20:
        f4:c5:ea:a1:f2:1b:24:a2:ad:b9:b6:c9:03:66:03:dd:1b:41:
        f2:21:c7:ed:31:21:63:21:56:2d:a1:5d:31:cd:91:85:8f:ef:
        a3:84:b0:34:59:ba:30:2b:da:ec:c5:55:8e:5d:c9:a6:da:e4:
        24:9e:76:b5:ca:83:46:fb:91:33:43:26:53:93:5c:2f:52:c0:
        23:ce:63:3f:ab:1e:46:89:cf:6c:3c:a4:26:80:76:b1:d6:85:
        d6:84:9a:db:6b:93:fe:3b:51:1a:2e:b8:8b:c7:76:c0:e1:bd:
        2d:cb:1d:de:fe:13:a4:57:18:76:b9:76:dd:5a:6f:79:49:cf:
        ed:44:b8:4b:97:ac:81:87:55:1c:0b:30:3b:84:ae:27:8a:97:
        bc:72:c8:96:65:f1:bd:63:85:0e:4c:d7:9d:4e:e4:9d:4d:b0:
        a1:8c:13:53:6a:a3:ba:cf:eb:ae:08:28:b2:5a:ec:0a:f4:e0:
        23:14:2f:c0:35:8b:da:a4:31:42:61:c9:2d:f9:57:3e:16:36:
        09:6c:59:37:3b:2e:06:2c:ee:9b:2c:66:fd:38:92:47:8c:2e:
        b8:5e:7c:19

		

Now leave the $HOME/ssl/ directory:

popd

Deploying our Keycloak instance with Podman

Set an initial administrator username and password. You should replace the examples below with more secure defaults:

			
export KEYCLOAK_ADMIN="admin" # replace me!
export KEYCLOAK_ADMIN_PASSWORD="P@ssw0rd" # replace me!

Now fetch the latest Keycloak image from Quay.io and run it:

			
podman run \
    --rm \
    -d \
    -p 8443:8443 \
    -e KEYCLOAK_ADMIN="${KEYCLOAK_ADMIN}" \
    -e KEYCLOAK_ADMIN_PASSWORD="${KEYCLOAK_ADMIN_PASSWORD}" \
    -v "$HOME/ssl/:/ssl/:U,Z,ro" \
    --name keycloak \
    quay.io/keycloak/keycloak:latest \
    start \
    --hostname=localhost \
    --https-certificate-file=/ssl/localhost.crt \
    --https-certificate-key-file=/ssl/localhost.key

		

Our Keycloak instance binds to port 8443 and uses the certificate and key we generated in a previous step for SSL/TLS encryption, bind-mounted into the container read-only with the correct ownership and SELinux context (U,Z,ro).

Visit https://localhost:8443/ in a web browser and click on “Administration Console” which presents a login portal. Enter your administrator credentials in the login portal and click “Sign In” which brings us to the administrator console.

Keycloak master realm administrator console

Notice that we are in the master realm. The master realm exists by default in Keycloak and should only be used to manage other realms. A realm in Keycloak is a tenant which represents different things depending on the context:

In a service provider context where a Keycloak instance is accessible to enterprise customers as a managed service, a tenant is an independent organization with its own identity and access management (IAM)
In an organizational context where a Keycloak instance is deployed for internal usage within the organization, a tenant is a department or organizational unit (OU) sharing a common base IdP such as the top-level corporate Active Directory domain

Creating a realm for our organization

Let’s create a realm for our example.com organization.

Click on the dropdown to the top left and select “Create realm”.

Fill in example-com as the realm name and click “Create”.

Observe that we are now in the administration console for the example-com realm.

example-com realm administration console

Enforcing mandatory MFA for all users in our realm

MFA is the practice of adding a second layer of authentication and is widely employed by organizations. Instead of solely verifying what a person “knows” (the password), MFA also verifies what a person “has” to gain confidence in securely associating a natural entity with a digital identity, such as a time-based one-time password (TOTP) generated by a mobile hardware or software device. This provides an additional layer of protection for accounts which have their password compromised, though an account with a compromised password should change their password immediately in any case.

Keycloak supports enforcing mandatory MFA at both the user and realm level. Suppose our organization’s IT security policy mandates that all corporate accounts be secured via MFA. This policy can then be satisfied by enabling OTP as a required action at the realm level. We’ll also enforce that passwords be changed at initial login so only the account owner knows their own password.

Navigate to the menu to the left and click “Authentication”, then select “Required actions” and ensure both “Configure OTP” and “Update Password” are enabled as default actions.

Creating an account for John Doe

John Doe has just joined our organization as a developer. As part of the onboarding process, let’s create an account for John Doe.

Select “Users > Add User”.

Fill in the username johndoe and any other fields as desired, then click “Create”.

Navigate to the “Credentials” tab and click “Set password”.

Fill in a suitable initial password for John Doe and click “Save”.

Click “Save password” to confirm the action.

Congratulations, you have created an account for John Doe and set an initial password!

Creating a client for the Keycloak SPA demo application

Platforms and applications integrating with Keycloak do so via clients. A client represents an application from Keycloak’s perspective.

Let’s test our Keycloak instance using an application. Fortunately, the Keycloak project conveniently provides a demo application so we don’t have to write our own. But first, we need to create a client for the app.

Select “Clients > Create client”.

Preserve the client type as “OpenID Connect” and fill in the client ID as keycloak-spa, then click “Next”.

Ensure “Standard flow” is checked and click “Next”.

example-com client application capability config

Now fill in the following details and click “Save”:

Valid redirect URLs: https://www.keycloak.org/app/*
Web origins: https://www.keycloak.org (without the trailing slash)

Congratulations, you have set up your first client on Keycloak!

Installing the Authenticator MFA app on Flathub

Before we proceed to log in as John Doe, remember that mandatory MFA is enabled so we’ll need an app to set it up during our first login.

Keycloak explicitly supports the following TOTP applications which can be installed on a mobile device:

But for a change, let’s see a TOTP app available on Fedora Linux. Authenticator is an open source GNOME application for generating TOTP codes available under the copyleft GPLv3+ license. It is part of the GNOME Circle, readily available on Flathub and integrates well with Fedora Workstation. Just make sure that Flathub is enabled as a repository source in GNOME Software, then find and install the app.

GNOME Software - FlatHub source repository

Logging in as John Doe with SSO

Now visit the demo application at https://www.keycloak.org/app/, fill in the configuration and click “Save”:

Keycloak URL: https://localhost:8443 (without the trailing slash)
Realm: example-com
Client: keycloak-spa

Click “Sign in”.

Fill in John Doe’s username and initial password, then click “Sign in”.

You will be prompted to configure MFA authentication based on TOTP. Open your Authenticator app, scan the QR code to add the application, then fill in the generated TOTP password and name the MFA device “Authenticator”. Click “Submit”.

Reset the initial password and ensure it is strong and secure, then click “Submit”.

Congratulations, you are now logged in as John Doe in the Keycloak SPA demo app via your Keycloak instance!

Cleaning up

Stop the Keycloak instance, then delete the SSL/TLS certificates created earlier. The Keycloak container was configured to be deleted automatically once stopped via the --rm flag.

			
podman stop keycloak
sudo rm -rf "$HOME/ssl/"

Concluding remarks and going further

Keycloak is an open source, enterprise-ready IdP supporting the most widely used authentication protocols. This enables it to readily integrate with a wide range of applications and platforms, including enterprise-grade orchestration platforms such as Kubernetes. Furthermore, it can be readily deployed to a wide range of environments, including on traditional servers / VMs, as a standalone container with Podman (and Docker), and on container orchestration platforms such as Kubernetes and OpenShift.

We have barely scratched the surface of Keycloak configuration in this lab. Keycloak can act directly as an IdP backend as in this lab, or act as a connector to existing enterprise IdP backends such as LDAP and Active Directory. Furthermore, by associating users with groups and roles, platforms integrating with Keycloak for authentication can apply role-based access control (RBAC) to enterprise identities for a scalable method of managing access to resources in a centralized manner.

+ Identity and Access Management, Security

Running workloads at the edge with MicroK8s

2024-02-13

The source code for this lab exercise is available on GitHub.

Edge computing is an extension of cloud computing and broadly refers to the paradigm where certain computational and data processing tasks are delegated from the cloud to devices residing at the network edge, close to the source of data being collected. This obviates the need for such edge nodes to forward the collected data to the cloud for central processing and decision making, greatly reducing the network traffic between cloud and edge and enabling edge nodes to make decisions autonomously in near-real time based on the collected data, which may be desirable or even required for certain industrial use cases.

Edge nodes usually possess one or more of the following characteristics:

Resides at the network edge as opposed to within a cloud datacenter
Resides close to one or more sources of data being collected, e.g. via sensors
Is resource constrained compared to enterprise server hardware
May suffer from an unstable, intermittent or nonexistent Internet connection
Are able or expected to process and respond to locally collected data in an autonomous fashion

MicroK8s is a lightweight, batteries included Kubernetes distribution by Canonical designed for running edge workloads which also happens to be developer-friendly and a great choice for building your own homelab. The following lab covers how to install and run MicroK8s on your own edge node running Ubuntu 22.04 LTS, deploy the NGINX web service and exposing your NGINX website to the Internet with SSL/TLS enabled using AWS resources included within the Free Tier.

Lab: Exposing your MicroK8s homelab to the Internet

Prerequisites

A basic understanding of Linux and Kubernetes is assumed. If not, consider enrolling in the following online, self-paced courses on edX offered by the Linux Foundation at no cost:

Setting up your environment

Apart from your laptop, desktop or workstation (any laptop will do), you’ll need:

An AWS account
An edge node running Ubuntu 22.04 LTS with at least 2 vCPUs, 4G memory and 20G storage

Some considerations:

Windows users may wish to complete the lab in a Unix/Linux environment within WSL2
The lab should work on other clouds such as Azure or Alibaba Cloud as well, but will require manual provisioning and configuration of cloud resources since as VM instances and security groups
The edge node running Ubuntu 22.04 LTS could be a spare laptop or Raspberry Pi (most exciting), a VM running on your own laptop with VirtualBox, VMware etc. (meh) or a cloud instance in a private subnet with outbound Internet connectivity (kind of defeats the purpose of this lab)

Generating an SSH key pair for our EC2 instance

Generate a dedicated SSH key pair for our Amazon EC2 instance which will be used both for connecting to our instance and creating an SSH reverse tunnel for forwarding HTTP(s) traffic to our MicroK8s edge node in a later step.

N.B. You are strongly advised to generate a dedicated SSH key pair for this lab even if you already have a default SSH key pair for security reasons.

ssh-keygen

Save the private key as $HOME/.ssh/microk8s-homelab.pem and the public key as $HOME/.ssh/microk8s-homelab.pub.

Setting up your AWS account

You’ll need to set up your AWS account and create an IAM administrator, then generate an access key and secret key for your IAM administrator for setting up AWS CLI v2 in our next step. Consult the official AWS documentation if in doubt.

Setting up AWS CLI v2

First make sure $HOME/.local/bin/ exists and is in your PATH so sudo is not required for installing the various command-line tools in this lab on your local laptop / desktop / workstation.

			
mkdir -p "$HOME/.local/bin/"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> "$HOME/.bashrc"
source "$HOME/.bashrc"

Now download AWS CLI v2 from the official website and install it using the provided script:

			
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -n awscliv2.zip
./aws/install --bin-dir "$HOME/.local/bin/" --install-dir "$HOME/.local/aws-cli/"

You’ll also need to configure AWS CLI v2 with your access and secret keys when prompted:

aws configure

Confirm that the setup is functional:

aws ec2 describe-instances

Sample output:

			
{
    "Reservations": []
}

Installing OpenTofu

OpenTofu is an open source infrastructure-as-code (IaC) tool that enables us to provision and manage AWS cloud resources in a completely automated manner, sparing us the effort of clicking through the AWS web console.

Install OpenTofu from the official release binaries. The latest version is 1.6.1 at the time of writing.

			
wget https://github.com/opentofu/opentofu/releases/download/v1.6.1/tofu_1.6.1_linux_amd64.zip
unzip -n tofu_1.6.1_linux_amd64.zip
mv tofu "$HOME/.local/bin/"

Confirm that OpenTofu is correctly installed.

tofu -version

Sample output:

			
OpenTofu v1.6.1
on linux_amd64

Deploying the automation stack

Clone the project repository and make it your working directory:

			
git clone https://github.com/DonaldKellett/microk8s-homelab.git
cd microk8s-homelab/

Set our cloud provider to AWS and fill in a valid, working email address for receiving Let’s Encrypt notifications in a later step; otherwise Let’s Encrypt will refuse to issue the required SSL/TLS certificates for securing our website.

			
export CLOUD_PROVIDER="aws"
export TF_VAR_email="webmaster@example.com" # replace me!

Now apply our OpenTofu configuration which creates the AWS cloud resources on our behalf.

			
tofu -chdir="opentofu/${CLOUD_PROVIDER}/" init
tofu -chdir="opentofu/${CLOUD_PROVIDER}/" apply

Answer yes when prompted and make note of the IP address printed to the console – this is the public IP address of our EC2 instance.

Preparing our EC2 instance for reverse tunneling

Now log into our EC2 instance. Remember to use the dedicated SSH key pair we generated earlier. Here’s what your SSH command should look like – replace x.x.x.x with the public IP address we noted earlier:

			
export MICROK8S_SSH_PRIVKEY="$HOME/.ssh/microk8s-homelab.pem" # replace me!
export MICROK8S_PROXY_IP="x.x.x.x" # replace me!
ssh -i "${MICROK8S_SSH_PRIVKEY}" ubuntu@"${MICROK8S_PROXY_IP}"

Run the remaining commands in this section on our EC2 instance.

Copy the SSH authorized keys to the root user. This is required for proxying the standard HTTP(s) ports (80/tcp and 443/tcp respectively) to our MicroK8s edge node via SSH reverse tunneling as non-root users cannot bind to privileged ports by default.

			
sudo mkdir -p /root/.ssh/
sudo cp "$HOME/.ssh/authorized_keys" /root/.ssh/authorized_keys
sudo chmod 700 /root/.ssh/
sudo chmod 600 /root/.ssh/authorized_keys

Now set the appropriate sshd configuration in /etc/ssh/sshd_config and restart the sshd service:

			
cat << EOF | sudo tee -a /etc/ssh/sshd_config
GatewayPorts yes
PermitRootLogin prohibit-password
EOF
sudo systemctl restart sshd.service

		

Ensure that the sshd service is active and running:

systemctl status sshd.service --full --no-pager

Sample output:

			
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/ssh.service.d
             └─ec2-instance-connect.conf
     Active: active (running) since Tue 2024-02-13 05:36:01 UTC; 1min 1s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 1370 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 1371 (sshd)
      Tasks: 1 (limit: 1091)
     Memory: 1.7M
        CPU: 24ms
     CGroup: /system.slice/ssh.service
             └─1371 "sshd: /usr/sbin/sshd -D -o AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys %u %f -o AuthorizedKeysCommandUser ec2-instance-connect [listener] 0 of 10-100 startups"
Feb 13 05:36:01 ip-10-0-1-12 systemd[1]: Starting OpenBSD Secure Shell server...
Feb 13 05:36:01 ip-10-0-1-12 sshd[1371]: Server listening on 0.0.0.0 port 22.
Feb 13 05:36:01 ip-10-0-1-12 sshd[1371]: Server listening on :: port 22.
Feb 13 05:36:01 ip-10-0-1-12 systemd[1]: Started OpenBSD Secure Shell server.

		

Our EC2 instance is now set up – log out of the EC2 instance and return to our laptop session.

Copying artifacts to our MicroK8s edge node

The SSH private key for connecting to our EC2 instance must be copied to our edge node for establishing the SSH reverse tunnel. The commands below assume you have SSH access to your edge node from your laptop – remember to replace the login username MICROK8S_USER and edge node IP MICROK8S_IP accordingly.

			
export MICROK8S_USER="ubuntu" # replace me!
export MICROK8S_IP="x.x.x.x" # replace me!
export MICROK8S_SSH_PRIVKEY="$HOME/.ssh/microk8s-homelab.pem" # replace me!
scp "${MICROK8S_SSH_PRIVKEY}" "${MICROK8S_USER}@${MICROK8S_IP}:~/microk8s-homelab.pem"

Additionally, when we created the AWS resources with OpenTofu, two Kubernetes manifests were also auto-generated under the manifests/ directory:

letsencrypt-clusterissuer.yaml: Let’s Encrypt ClusterIssuer for cert-manager which automates the process of obtaining an SSL/TLS certificate from Let’s Encrypt
nginx-ingress.yaml: Kubernetes Ingress for our NGINX web service. This enables us to visit our website via a domain name instead of entering the IP address directly

Here’s what the manifests look like – yours will be slightly different since the EC2 public IP addresses are different:

			
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: donaldsebleung@gmail.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: public
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  creationTimestamp: null
  name: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  rules:
  - host: nginx.16.163.22.52.sslip.io
    http:
      paths:
      - backend:
          service:
            name: nginx
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - nginx.16.163.22.52.sslip.io
    secretName: nginx-cert
status:
  loadBalancer: {}

		

Notice the domain name nginx.16.163.22.52.sslip.io (this depends on the IP address). This is a subdomain under sslip.io, a public DNS service that returns the same IP address given a domain name with an embedded IP address. Using this DNS service enables us to request Let’s Encrypt certificates without purchasing our own domain name.

Copy these manifests to our edge node:

			
scp manifests/nginx-ingress.yaml "${MICROK8S_USER}@${MICROK8S_IP}:~/nginx-ingress.yaml"
scp manifests/letsencrypt-clusterissuer.yaml "${MICROK8S_USER}@${MICROK8S_IP}:~/letsencrypt-clusterissuer.yaml"

Now SSH into our edge node and run the remaining commands on our edge node unless otherwise instructed.

Installing MicroK8s on our edge node

MicroK8s can be installed as a Snap on Ubuntu with a single command (this may take a while depending on network bandwidth):

sudo snap install microk8s --classic

Now add the current user to the microk8s group:

sudo usermod -aG microk8s "${USER}"

Log out and in again for group membership to take effect.

Wait for MicroK8s to become ready – again, this may take a few minutes:

microk8s status --wait-ready

Sample output:

			
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
  disabled:
    cert-manager         # (core) Cloud native certificate management
    cis-hardening        # (core) Apply CIS K8s hardening
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    rook-ceph            # (core) Distributed Ceph storage using Rook
    storage              # (core) Alias to hostpath-storage add-on, deprecated

		

Creating our SSH reverse tunnel with AutoSSH

Before creating our SSH reverse tunnel, enable the ingress addon which allows external users to access services running on our MicroK8s cluster. Addons are a unique feature of MicroK8s which enable us to start using common Kubernetes extensions and middleware with a single command.

microk8s enable ingress

Sample output:

			
Infer repository core for addon ingress
Enabling Ingress
ingressclass.networking.k8s.io/public created
ingressclass.networking.k8s.io/nginx created
namespace/ingress created
serviceaccount/nginx-ingress-microk8s-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
configmap/nginx-load-balancer-microk8s-conf created
configmap/nginx-ingress-tcp-microk8s-conf created
configmap/nginx-ingress-udp-microk8s-conf created
daemonset.apps/nginx-ingress-microk8s-controller created
Ingress is enabled

		

Install AutoSSH for creating SSH connections that persist across reboots, a requirement for our SSH reverse tunnel:

sudo apt update && sudo apt install -y autossh

Now create the corresponding AutoSSH systemd service – replace x.x.x.x with the EC2 public IP address we noted earlier:

			
export MICROK8S_PROXY_IP="x.x.x.x" # replace me!
cat << EOF | sudo tee /etc/systemd/system/autossh.service
[Unit]
Description=SSH reverse tunnel for ingress
After=network.target
[Service]
User=${USER}
ExecStart=/usr/bin/autossh \
    -i "$HOME/microk8s-homelab.pem" \
    -o "ServerAliveInterval 60" \
    -o "ServerAliveCountMax 5" \
    -o "StrictHostKeyChecking no" \
    -o "UserKnownHostsFile /dev/null" \
    -nNT \
    -R 0.0.0.0:80:localhost:80 \
    -R 0.0.0.0:443:localhost:443 \
    root@"${MICROK8S_PROXY_IP}"
[Install]
WantedBy=multi-user.target
EOF

		

Start and enable the AutoSSH service:

			
sudo systemctl daemon-reload
sudo systemctl enable --now autossh.service

Ensure the AutoSSH service is active and running:

systemctl status autossh.service --full --no-pager

Sample output:

			
● autossh.service - SSH reverse tunnel for ingress
     Loaded: loaded (/etc/systemd/system/autossh.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-02-13 06:27:25 UTC; 4s ago
   Main PID: 24270 (autossh)
      Tasks: 2 (limit: 4558)
     Memory: 3.1M
        CPU: 30ms
     CGroup: /system.slice/autossh.service
             ├─24270 /usr/lib/autossh/autossh -i /home/dsleung/microk8s-homelab.pem -o "ServerAliveInterval 60" -o "ServerAliveCountMax 5" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -nNT -R 0.0.0.0:80:localhost:80 -R 0.0.0.0:443:localhost:443 root@16.163.22.52
             └─24281 /usr/bin/ssh -L 62034:127.0.0.1:62034 -R 62034:127.0.0.1:62035 -i /home/dsleung/microk8s-homelab.pem -o "ServerAliveInterval 60" -o "ServerAliveCountMax 5" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -nNT -R 0.0.0.0:80:localhost:80 -R 0.0.0.0:443:localhost:443 root@16.163.22.52
Feb 13 06:27:25 microk8s systemd[1]: Started SSH reverse tunnel for ingress.
Feb 13 06:27:25 microk8s autossh[24270]: starting ssh (count 1)
Feb 13 06:27:25 microk8s autossh[24270]: ssh child pid is 24281
Feb 13 06:27:25 microk8s autossh[24281]: Warning: Permanently added '16.163.22.52' (ED25519) to the list of known hosts.

		

Now visit https://nginx.16.163.22.52.sslip.io/ in a web browser. Remember to replace 16.163.22.52 with your EC2 public IP address. You will receive a certificate warning from your browser which you should ignore and proceed with visiting the website, for which you should be greeted with a 404 page.

This is expected since we have not set up our website yet, but it tells us that our Kubernetes ingress running on our edge node is reachable from the public Internet via our EC2 cloud instance thanks to the SSH reverse tunnel – how cool is that? 😀

Setting up our website with SSL/TLS using cert-manager and Let’s Encrypt

cert-manager provides fully automated X.509 certificate management for Kubernetes and OpenShift via custom resource definitions (CRDs), obviating the need for manually running arcane openssl commands. It is available as a MicroK8s addon so can be enabled with a single command:

microk8s enable cert-manager

Sample output:

			
Infer repository core for addon cert-manager
Enable DNS addon
Infer repository core for addon dns
Addon core/dns is already enabled
Enabling cert-manager
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
configmap/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
Waiting for cert-manager to be ready.
...ready
Enabled cert-manager
===========================
Cert-manager is installed. As a next step, try creating a ClusterIssuer
for Let's Encrypt by creating the following resource:
$ microk8s kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: me@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: public
EOF
Then, you can create an ingress to expose 'my-service:80' on 'https://my-service.example.com' with:
$ microk8s enable ingress
$ microk8s kubectl create ingress my-ingress \
    --annotation cert-manager.io/cluster-issuer=letsencrypt \
    --rule 'my-service.example.com/*=my-service:80,tls=my-service-tls'

		

Before we proceed, here’s a useful tip for the microk8s kubectl commands to follow. Make liberal use of tab auto-completion – for example, if you type:

microk8s kubectl cr<TAB>

Where <TAB> represents a literal Tab keypress, you should get:

microk8s kubectl create

This is especially useful for kubectl commands which are often lengthy and verbose by saving you some precious typing time.

Let’s Encrypt is a certificate authority (CA) which issues SSL/TLS certificates at no cost, making it an attractive option for securing websites with HTTPs.

Create the Let’s Encrypt ClusterIssuer with the manifest we saw earlier:

microk8s kubectl apply -f "$HOME/letsencrypt-clusterissuer.yaml"

Now wait for the ClusterIssuer to become ready:

			
microk8s kubectl wait \
    --for=condition=Ready \
    clusterissuers.cert-manager.io \
    letsencrypt \
    --timeout=180s

		

Sample output:

clusterissuer.cert-manager.io/letsencrypt condition met

Create an NGINX deployment for our website and expose it as a Kubernetes service:

			
microk8s kubectl create deploy nginx \
    --image=nginx \
    --replicas=2 \
    --port=80
microk8s kubectl expose deploy nginx

		

Now create an Ingress resource for our website using the manifest we saw earlier:

microk8s kubectl apply -f "$HOME/nginx-ingress.yaml"

Wait for the nginx-cert certificate to become ready:

			
microk8s kubectl wait \
    --for=condition=Ready \
    certificates.cert-manager.io \
    nginx-cert \
    --timeout=180s

		

Sample output:

certificate.cert-manager.io/nginx-cert condition met

Now open a new private window and visit our website again. Notice the following:

We no longer receive a certificate warning since the Let’s Encrypt certificate is trusted and secure
We are greeted with the NGINX welcome page which indicates our website is functional
If we view the website certificate, we see that it is officially signed by Let’s Encrypt

NGINX website certificate signed by Let's Encrypt

Congratulations! You have successfully set up a MicroK8s edge node, deployed an NGINX web service to MicroK8s and exposed it to the Internet securely with SSL/TLS encryption using an SSH reverse tunnel, Kubernetes ingress, cert-manager and Let’s Encrypt.

Log out of the MicroK8s edge node and return to your laptop session.

Tearing down

Tear down the AWS infrastructure provisioned for this lab to save costs.

tofu -chdir="opentofu/${CLOUD_PROVIDER}/" destroy

Concluding remarks and going further

We saw how MicroK8s is a lightweight, batteries included Kubernetes distribution that “just works” for edge computing use cases, developer environments and homelabs alike. With the unique “addon” approach, many common Kubernetes extensions and middleware can be enabled on MicroK8s with a single microk8s enable command and it is even possible to develop and distribute your own addons since MicroK8s 1.24. Furthermore, we saw how cert-manager automates the management of SSL/TLS certificates through Kubernetes custom resources so you don’t have to type in arcane openssl commands or perform step-by-step domain verification manually.

I hope you enjoyed this lab and stay tuned for updates 😉

+ Cloud, Edge, Kubernetes

Exploring OpenShift with CRC

2024-01-14

OpenShift Container Platform (OCP), otherwise known as just OpenShift, is a comprehensive, feature-complete enterprise PaaS offering by Red Hat built on top of Kubernetes, available both as a fully managed service on popular public cloud platforms such as AWS (ROSA) and as an internal developer platform (IDP) to be deployed on-premises on existing private cloud infrastructure, as VMs or on bare metal.

Compared to Kubernetes, OpenShift:

Is an out-of-the-box IDP as opposed to an infrastructure layer for building your own IDP
Places a strong emphasis on the developer experience and enhancing developer productivity
Officially supports automated provisioning of infrastructure on supported platforms as part of the cluster creation process, as opposed to manual cluster creation with kubeadm (optionally) relying on third party tooling and integration for automation
Is an open-core commercial offering, as opposed to a fully open source project
Is opinionated, as opposed to flexible and customizable

However, due to its comprehensive feature set and extensive automation, a production-grade OpenShift cluster has non-trivial resource requirements, making it out of reach for individual developers and home lab enthusiasts wanting to try out OpenShift locally or on commodity hardware such as various editions of the Raspberry Pi.

Fortunately, just as projects like kind and Minikube enable developers to spin up a local Kubernetes environment in no time, CRC, also known as OpenShift Local and a recursive acronym for “CRC – Runs Containers”, offers developers a local OpenShift environment by means of a pre-configured VM similar to how Minikube works under the hood.

In the lab to follow, we will provision a single-node OpenShift cluster using CRC, convince ourselves that OpenShift is in fact Kubernetes (but not just) and briefly explore what OpenShift has to offer. The good news – all you need is your own laptop (or desktop / workstation) to follow through!

Lab: Provisioning a single-node OKD cluster using CRC

OKD is the community distribution of Kubernetes that powers Red Hat OpenShift and is available under the permissive Apache 2.0 open source license, in the same spirit as upstream Kubernetes. It is also known as OpenShift Origin and rumors are that its acronym stands for “Origin Kubernetes Distribution”, though you will not find this anywhere in the official documentation. Apart from the lack of commercial support and access to Red Hat certified Operators (more on that later), OKD is otherwise functionally identical to OCP which is the commercial version of OpenShift.

While CRC does support the creation of a local OCP cluster, we will stick to OKD since (1) it allows us to focus on the open source ecosystem around OpenShift and (2) we won’t be getting any commercial support from Red Hat for OpenShift Local anyway 😛

For the sake of simplicity, we will refer to OKD as simply OpenShift going forward where the context allow, except when it might lead to confusion with the commercial edition (OCP).

Prerequisites

A basic understanding of Kubernetes is assumed. If not, consider enrolling in LFS158x: Introduction to Kubernetes, a self-paced online course offered by the Linux Foundation on edX at no cost.

Setting up your environment

You’ll need a laptop (or desktop / workstation) fulfilling the minimum resource requirements outlined below:

4 physical CPU cores (8 vCPU cores)
16 GiB of RAM
At least 35 GiB of available storage

Being a feature-complete IDP, OpenShift requires more resources compared to Kubernetes so unfortunately a decade-old laptop with only 4 GiB of RAM or a MacBook Air will not suffice (-:

All the major desktop operating systems are supported, including Windows, macOS and (popular distributions of) Linux, though note that some commands might require minor modification on Windows.

Now download the latest version of CRC and place it under your PATH – for example, on Linux, one might run the following commands:

			
wget https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.31.0/crc-linux-amd64.tar.xz
tar xvf crc-linux-amd64.tar.xz
mkdir -p "$HOME/.local/bin/"
install crc-linux-2.31.0-amd64/crc "$HOME/.local/bin/crc"

Verify the installation now:

crc version

Sample output:

			
CRC version: 2.31.0+6d23b6
OpenShift version: 4.14.7
Podman version: 4.4.4

Spinning up an OpenShift Local cluster

Set OKD as the desired OpenShift distribution:

crc config set preset okd

Now run the setup command, which may take a few minutes on initial execution:

crc setup

If crc setup exits with an error, it should tell you what dependencies you should manually install and/or configure. In that case, follow the instructions provided in the error message to resolve the issue and try again.

Now spin up our OpenShift cluster, which may take some 15-30 minutes – grab a cup of ~~coffee~~ tea and wait patiently while the cluster bootstraps and stabilizes itself:

crc start

Sample output (kubeadmin password elided for security reasons):

			
INFO Using bundle path /home/donaldsebleung/.crc/cache/crc_okd_libvirt_4.14.0-0.okd-2023-12-01-225814_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
INFO Checking if crc executable symlink exists    
INFO Checking minimum RAM requirements            
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking crc daemon systemd socket units     
INFO Checking if systemd-networkd is running      
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if dnsmasq configurations file exist for NetworkManager 
INFO Checking if the systemd-resolved service is running 
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
INFO Checking if libvirt 'crc' network is available 
INFO Checking if libvirt 'crc' network is active  
INFO Loading bundle: crc_okd_libvirt_4.14.0-0.okd-2023-12-01-225814_amd64... 
INFO Creating CRC VM for OKD 4.14.0-0.okd-2023-12-01-225814... 
INFO Generating new SSH key pair...               
INFO Generating new password for the kubeadmin user 
INFO Starting CRC VM for okd 4.14.0-0.okd-2023-12-01-225814... 
INFO CRC instance is running with IP 192.168.130.11 
INFO CRC VM is running                            
INFO Updating authorized keys...                  
INFO Configuring shared directories               
INFO Check internal and public DNS query...       
INFO Check DNS query from host...                 
INFO Verifying validity of the kubelet certificates... 
INFO Starting kubelet service                     
INFO Kubelet client certificate has expired, renewing it... [will take up to 8 minutes] 
INFO Kubelet serving certificate has expired, waiting for automatic renewal... [will take up to 8 minutes] 
INFO Waiting for kube-apiserver availability... [takes around 2min] 
INFO Adding user's pull secret to the cluster...  
INFO Updating SSH key to machine config resource... 
INFO Waiting until the user's pull secret is written to the instance disk... 
INFO Changing the password for the kubeadmin user 
INFO Updating cluster ID...                       
INFO Updating root CA cert to admin-kubeconfig-client-ca configmap... 
INFO Starting okd instance... [waiting for the cluster to stabilize] 
INFO 2 operators are progressing: kube-apiserver, openshift-controller-manager 
INFO 2 operators are progressing: kube-apiserver, openshift-controller-manager 
INFO Operator kube-apiserver is progressing       
INFO Operator kube-apiserver is progressing       
INFO Operator authentication is not yet available 
INFO Operator authentication is not yet available 
INFO Operator authentication is degraded          
INFO All operators are available. Ensuring stability... 
INFO Operators are stable (2/3)...                
INFO Operators are stable (3/3)...                
INFO Adding crc-admin and crc-developer contexts to kubeconfig... 
Started the OpenShift cluster.
The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing
Log in as administrator:
  Username: kubeadmin
  Password: XXXXX-XXXXX-XXXXX-XXXXX
Log in as user:
  Username: developer
  Password: developer
Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443
NOTE:
This cluster was built from OKD - The Community Distribution of Kubernetes that powers Red Hat OpenShift.
If you find an issue, please report it at https://github.com/openshift/okd

		

Congratulations – you now have a working OpenShift cluster running locally! It’s time to explore our newly created OpenShift environment 🙂

Take note of the kubeadmin password shown above as we will be using it later.

Exploring our cluster with the `kubectl` `oc` command line

oc is the official command-line tool for interacting with OpenShift clusters, much like kubectl is to Kubernetes.

While you can fetch a copy of oc binary from the public OpenShift mirrors, CRC comes with a bundled version of oc itself – just evaluate the output of crc oc-env to start using oc:

eval $(crc oc-env)

Let’s check which version of oc and OpenShift is installed:

oc version

Sample output:

			
Client Version: 4.14.0-0.okd-2023-12-01-225814
Kustomize Version: v5.0.1
Server Version: 4.14.0-0.okd-2023-12-01-225814
Kubernetes Version: v1.27.1-3443+19254894103e33-dirty

We see that our OpenShift client and server versions are at 4.14, corresponding to a Kubernetes version of 1.27. As mentioned earlier, OpenShift is Kubernetes at its core, but not just 😉

Let’s play with a few oc commands to really convince ourselves that OpenShift is Kubernetes.

View the nodes in our cluster – there should be only 1:

oc get nodes

Sample output:

			
NAME                 STATUS   ROLES                         AGE   VERSION
crc-d4g2s-master-0   Ready    control-plane,master,worker   40d   v1.27.6+d548052

Let’s take a closer look at our node – your node name may be different so substitue crc-d4g2s-master-0 with the name reported in your terminal output in the previous step:

oc describe node crc-d4g2s-master-0

Sample output:

			
Name:               crc-d4g2s-master-0
Roles:              control-plane,master,worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=crc-d4g2s-master-0
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node-role.kubernetes.io/master=
                    node-role.kubernetes.io/worker=
                    node.openshift.io/os_id=fedora
                    topology.hostpath.csi/node=crc-d4g2s-master-0
Annotations:        csi.volume.kubernetes.io/nodeid: {"kubevirt.io.hostpath-provisioner":"crc-d4g2s-master-0"}
                    machine.openshift.io/machine: openshift-machine-api/crc-d4g2s-master-0
                    machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica
                    machineconfiguration.openshift.io/currentConfig: rendered-master-937128c862a772f4cab37531e27977f6
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-937128c862a772f4cab37531e27977f6
                    machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-master-937128c862a772f4cab37531e27977f6
                    machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-master-937128c862a772f4cab37531e27977f6
                    machineconfiguration.openshift.io/lastSyncedControllerConfigResourceVersion: 34100
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 03 Dec 2023 12:54:31 +0800
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  crc-d4g2s-master-0
  AcquireTime:     <unset>
  RenewTime:       Fri, 12 Jan 2024 22:56:16 +0800
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 12 Jan 2024 22:52:54 +0800   Fri, 12 Jan 2024 22:37:46 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 12 Jan 2024 22:52:54 +0800   Fri, 12 Jan 2024 22:37:46 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 12 Jan 2024 22:52:54 +0800   Fri, 12 Jan 2024 22:37:46 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 12 Jan 2024 22:52:54 +0800   Fri, 12 Jan 2024 22:38:58 +0800   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.126.11
  Hostname:    crc-d4g2s-master-0
Capacity:
  cpu:                4
  ephemeral-storage:  31914988Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             9137272Ki
  pods:               250
Allocatable:
  cpu:                3800m
  ephemeral-storage:  29045851293
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8676472Ki
  pods:               250
System Info:
  Machine ID:                                       57cc67c317aa4a1d92b0d8b9a789be8c
  System UUID:                                      b7504a6e-aafd-4203-964b-a3c196e3b82a
  Boot ID:                                          9adc8a51-44a3-4ab5-a963-8acf742b7db9
  Kernel Version:                                   6.5.5-200.fc38.x86_64
  OS Image:                                         Fedora CoreOS 38.20231027.3.2
  Operating System:                                 linux
  Architecture:                                     amd64
  Container Runtime Version:                        cri-o://1.27.0
  Kubelet Version:                                  v1.27.6+d548052
  Kube-Proxy Version:                               v1.27.6+d548052
Non-terminated Pods:                                (60 in total)
  Namespace                                         Name                                                       CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                                         ----                                                       ------------  ----------  ---------------  -------------  ---
  hostpath-provisioner                              csi-hostpathplugin-l756x                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         39d
  openshift-apiserver-operator                      openshift-apiserver-operator-6fb8688846-8zzt9              10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-apiserver                               apiserver-678745575d-s8gg2                                 110m (2%)     0 (0%)      250Mi (2%)       0 (0%)         13m
  openshift-authentication-operator                 authentication-operator-6667569b8d-2qfhd                   20m (0%)      0 (0%)      200Mi (2%)       0 (0%)         40d
  openshift-authentication                          oauth-openshift-579b5549b5-2vqfr                           10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         11m
  openshift-cluster-machine-approver                machine-approver-865888d45-qbqqb                           20m (0%)      0 (0%)      70Mi (0%)        0 (0%)         40d
  openshift-cluster-samples-operator                cluster-samples-operator-5f7d584d58-z7m4q                  20m (0%)      0 (0%)      100Mi (1%)       0 (0%)         40d
  openshift-cluster-version                         cluster-version-operator-7cc7d6cc78-cvf5x                  20m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-config-operator                         openshift-config-operator-77dd576b98-llq4n                 10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-console-operator                        console-operator-555f64dfdd-9k8td                          20m (0%)      0 (0%)      200Mi (2%)       0 (0%)         40d
  openshift-console                                 console-6679fcfc67-ttv8j                                   10m (0%)      0 (0%)      100Mi (1%)       0 (0%)         40d
  openshift-console                                 downloads-557648459f-xwnrb                                 10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-controller-manager-operator             openshift-controller-manager-operator-699898cf9c-vfp2t     10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-controller-manager                      controller-manager-84b6b4d58f-9xs6q                        100m (2%)     0 (0%)      100Mi (1%)       0 (0%)         12m
  openshift-dns-operator                            dns-operator-6d8b574c64-95wgs                              20m (0%)      0 (0%)      69Mi (0%)        0 (0%)         40d
  openshift-dns                                     dns-default-l9sgb                                          60m (1%)      0 (0%)      110Mi (1%)       0 (0%)         40d
  openshift-dns                                     node-resolver-f6sdd                                        5m (0%)       0 (0%)      21Mi (0%)        0 (0%)         40d
  openshift-etcd-operator                           etcd-operator-7885b9b76f-pc45h                             10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-etcd                                    etcd-crc-d4g2s-master-0                                    360m (9%)     0 (0%)      910Mi (10%)      0 (0%)         40d
  openshift-image-registry                          cluster-image-registry-operator-79775695c6-ph9gw           10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-image-registry                          image-registry-6cf94f6cf6-d6t82                            100m (2%)     0 (0%)      256Mi (3%)       0 (0%)         12m
  openshift-image-registry                          node-ca-k5w4n                                              10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         40d
  openshift-ingress-canary                          ingress-canary-mnzrz                                       10m (0%)      0 (0%)      20Mi (0%)        0 (0%)         40d
  openshift-ingress-operator                        ingress-operator-6f4d57d459-bpkb9                          20m (0%)      0 (0%)      96Mi (1%)        0 (0%)         40d
  openshift-ingress                                 router-default-5c59975b6f-gcths                            100m (2%)     0 (0%)      256Mi (3%)       0 (0%)         40d
  openshift-kube-apiserver-operator                 kube-apiserver-operator-9c4dc788b-687gt                    10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-kube-apiserver                          kube-apiserver-crc-d4g2s-master-0                          290m (7%)     0 (0%)      1224Mi (14%)     0 (0%)         10m
  openshift-kube-controller-manager-operator        kube-controller-manager-operator-6d696d7f5b-vlxh8          10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-kube-controller-manager                 kube-controller-manager-crc-d4g2s-master-0                 80m (2%)      0 (0%)      500Mi (5%)       0 (0%)         40d
  openshift-kube-scheduler-operator                 openshift-kube-scheduler-operator-6967cf8888-s7l9p         10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-kube-scheduler                          openshift-kube-scheduler-crc-d4g2s-master-0                25m (0%)      0 (0%)      150Mi (1%)       0 (0%)         40d
  openshift-kube-storage-version-migrator-operator  kube-storage-version-migrator-operator-7f56f455c7-9qm2x    10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-kube-storage-version-migrator           migrator-54754c959-rpbdh                                   10m (0%)      0 (0%)      200Mi (2%)       0 (0%)         40d
  openshift-machine-api                             control-plane-machine-set-operator-7d8877fb45-bfkcd        10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-machine-api                             machine-api-controllers-548c55cccd-x45wq                   70m (1%)      0 (0%)      140Mi (1%)       0 (0%)         40d
  openshift-machine-api                             machine-api-operator-659b96dd6f-b2pzw                      20m (0%)      0 (0%)      70Mi (0%)        0 (0%)         40d
  openshift-machine-config-operator                 machine-config-controller-794b56d4dc-xk7jc                 40m (1%)      0 (0%)      100Mi (1%)       0 (0%)         40d
  openshift-machine-config-operator                 machine-config-daemon-qgct8                                40m (1%)      0 (0%)      100Mi (1%)       0 (0%)         40d
  openshift-machine-config-operator                 machine-config-operator-84dc94fd78-9x5wj                   40m (1%)      0 (0%)      100Mi (1%)       0 (0%)         40d
  openshift-machine-config-operator                 machine-config-server-cvrfx                                20m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-marketplace                             community-operators-ccvhg                                  10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-marketplace                             marketplace-operator-6874dc8464-lmlvx                      1m (0%)       0 (0%)      5Mi (0%)         0 (0%)         40d
  openshift-multus                                  multus-4xhrh                                               10m (0%)      0 (0%)      65Mi (0%)        0 (0%)         40d
  openshift-multus                                  multus-additional-cni-plugins-62lw4                        10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         40d
  openshift-multus                                  multus-admission-controller-5f4676c68-29b2m                20m (0%)      0 (0%)      70Mi (0%)        0 (0%)         39d
  openshift-multus                                  network-metrics-daemon-kwd8t                               20m (0%)      0 (0%)      120Mi (1%)       0 (0%)         40d
  openshift-network-diagnostics                     network-check-source-7894f8cc69-88mhc                      10m (0%)      0 (0%)      40Mi (0%)        0 (0%)         40d
  openshift-network-diagnostics                     network-check-target-knh9x                                 10m (0%)      0 (0%)      15Mi (0%)        0 (0%)         40d
  openshift-network-node-identity                   network-node-identity-ffl7r                                20m (0%)      0 (0%)      100Mi (1%)       0 (0%)         40d
  openshift-network-operator                        network-operator-fd55479ff-6ld9p                           10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-oauth-apiserver                         apiserver-5d49456867-jpzrt                                 150m (3%)     0 (0%)      200Mi (2%)       0 (0%)         40d
  openshift-operator-lifecycle-manager              catalog-operator-fc994d6b6-tj8h5                           10m (0%)      0 (0%)      80Mi (0%)        0 (0%)         40d
  openshift-operator-lifecycle-manager              olm-operator-75bbbb5559-zft8l                              10m (0%)      0 (0%)      160Mi (1%)       0 (0%)         40d
  openshift-operator-lifecycle-manager              package-server-manager-7c55fcf5f7-4dqpc                    10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-operator-lifecycle-manager              packageserver-999d54647-66s5p                              10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         40d
  openshift-route-controller-manager                route-controller-manager-8559f9c7cb-trktg                  100m (2%)     0 (0%)      100Mi (1%)       0 (0%)         12m
  openshift-sdn                                     sdn-controller-b49gh                                       20m (0%)      0 (0%)      70Mi (0%)        0 (0%)         40d
  openshift-sdn                                     sdn-v64p2                                                  110m (2%)     0 (0%)      220Mi (2%)       0 (0%)         40d
  openshift-service-ca-operator                     service-ca-operator-56c579c485-cpwv7                       10m (0%)      0 (0%)      80Mi (0%)        0 (0%)         40d
  openshift-service-ca                              service-ca-6c5d767655-7xw57                                10m (0%)      0 (0%)      120Mi (1%)       0 (0%)         40d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                2321m (61%)   0 (0%)
  memory             7707Mi (90%)  0 (0%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-1Gi      0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
Events:
  Type    Reason                   Age                From             Message
  ----    ------                   ----               ----             -------
  Normal  NodeHasSufficientMemory  40d (x8 over 40d)  kubelet          Node crc-d4g2s-master-0 status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    40d (x8 over 40d)  kubelet          Node crc-d4g2s-master-0 status is now: NodeHasNoDiskPressure
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           40d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           39d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  NodeNotReady             39d                node-controller  Node crc-d4g2s-master-0 status is now: NodeNotReady
  Normal  Starting                 39d                kubelet          Starting kubelet.
  Normal  NodeAllocatableEnforced  39d                kubelet          Updated Node Allocatable limit across pods
  Normal  NodeHasSufficientPID     39d (x7 over 39d)  kubelet          Node crc-d4g2s-master-0 status is now: NodeHasSufficientPID
  Normal  NodeHasNoDiskPressure    39d (x8 over 39d)  kubelet          Node crc-d4g2s-master-0 status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientMemory  39d (x8 over 39d)  kubelet          Node crc-d4g2s-master-0 status is now: NodeHasSufficientMemory
  Normal  RegisteredNode           39d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           39d                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  NodeNotReady             39d                node-controller  Node crc-d4g2s-master-0 status is now: NodeNotReady
  Normal  RegisteredNode           18m                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller
  Normal  RegisteredNode           10m                node-controller  Node crc-d4g2s-master-0 event: Registered Node crc-d4g2s-master-0 in Controller

		

In particular, notice the OS image reported above under “System Info” is Fedora CoreOS 38.20231027.3.2. Fedora CoreOS, also known as FCOS, is a container-optimized OS under the Fedora project designed to run containerized workloads at scale – in particular, it is the only supported operating system for OKD control plane nodes, while OKD worker nodes can run Fedora Server as well. Since OKD only supports Fedora as the underlying operating system, it is opinionated in this sense compared to upstream Kubernetes which works on most Linux distributions and even Windows (albeit with a limited subset of features) for Kubernetes worker nodes. Not surprisingly, the commercial OCP distribution downstream from OKD requires Red Hat Enterprise Linux CoreOS (RHCOS) for its control plane nodes and optionally RHEL for its worker nodes, for which both OS editions are their respective downstreams from Fedora.

Now let’s see how many pods are running on our cluster:

oc get pods --all-namespaces --no-headers | wc -l

Sample output:

That’s a lot of pods on a freshly installed cluster! Compare this to a fresh Minikube cluster which has only 7 pods. But when you recall that OpenShift is a full-fledged IDP as opposed to a minimal infrastructure layer, everything suddenly makes sense 😉

View the available namespaces:

oc get namespaces

Sample output:

			
NAME                                               STATUS   AGE
default                                            Active   40d
hostpath-provisioner                               Active   39d
kube-node-lease                                    Active   40d
kube-public                                        Active   40d
kube-system                                        Active   40d
openshift                                          Active   40d
openshift-apiserver                                Active   40d
openshift-apiserver-operator                       Active   40d
openshift-authentication                           Active   40d
openshift-authentication-operator                  Active   40d
openshift-cloud-controller-manager                 Active   40d
openshift-cloud-controller-manager-operator        Active   40d
openshift-cloud-credential-operator                Active   40d
openshift-cloud-network-config-controller          Active   40d
openshift-cluster-machine-approver                 Active   40d
openshift-cluster-samples-operator                 Active   40d
openshift-cluster-storage-operator                 Active   40d
openshift-cluster-version                          Active   40d
openshift-config                                   Active   40d
openshift-config-managed                           Active   40d
openshift-config-operator                          Active   40d
openshift-console                                  Active   40d
openshift-console-operator                         Active   40d
openshift-console-user-settings                    Active   40d
openshift-controller-manager                       Active   40d
openshift-controller-manager-operator              Active   40d
openshift-dns                                      Active   40d
openshift-dns-operator                             Active   40d
openshift-etcd                                     Active   40d
openshift-etcd-operator                            Active   40d
openshift-host-network                             Active   40d
openshift-image-registry                           Active   40d
openshift-infra                                    Active   40d
openshift-ingress                                  Active   40d
openshift-ingress-canary                           Active   40d
openshift-ingress-operator                         Active   40d
openshift-kni-infra                                Active   40d
openshift-kube-apiserver                           Active   40d
openshift-kube-apiserver-operator                  Active   40d
openshift-kube-controller-manager                  Active   40d
openshift-kube-controller-manager-operator         Active   40d
openshift-kube-scheduler                           Active   40d
openshift-kube-scheduler-operator                  Active   40d
openshift-kube-storage-version-migrator            Active   40d
openshift-kube-storage-version-migrator-operator   Active   40d
openshift-machine-api                              Active   40d
openshift-machine-config-operator                  Active   40d
openshift-marketplace                              Active   40d
openshift-monitoring                               Active   40d
openshift-multus                                   Active   40d
openshift-network-diagnostics                      Active   40d
openshift-network-node-identity                    Active   40d
openshift-network-operator                         Active   40d
openshift-node                                     Active   40d
openshift-nutanix-infra                            Active   40d
openshift-oauth-apiserver                          Active   40d
openshift-openstack-infra                          Active   40d
openshift-operator-lifecycle-manager               Active   40d
openshift-operators                                Active   40d
openshift-ovirt-infra                              Active   40d
openshift-route-controller-manager                 Active   40d
openshift-sdn                                      Active   40d
openshift-service-ca                               Active   40d
openshift-service-ca-operator                      Active   40d
openshift-user-workload-monitoring                 Active   40d
openshift-vsphere-infra                            Active   40d

		

Again, that’s a lot. Apart from the Kubernetes system namespaces (default, kube-system, kube-node-lease, kube-public) and hostpath-provisioner which contains a CSI driver for dynamic provisioning of storage (think PVCs), the other 60 or so namespaces starting with openshift* are OpenShift-reserved namespaces for running critical OpenShift cluster components and services.

View the services in the default namespace:

oc get services

Sample output:

			
NAME         TYPE           CLUSTER-IP   EXTERNAL-IP                            PORT(S)   AGE
kubernetes   ClusterIP      10.217.4.1   <none>                                 443/TCP   40d
openshift    ExternalName   <none>       kubernetes.default.svc.cluster.local   <none>    40d

Not much different from a fresh Kubernetes install, except the addition of the openshift service which is an ExternalName pointing to the kubernetes service, i.e. a CNAME alias from the DNS perspective.

Let’s try out an NGINX workload. Create an NGINX deployment with 2 replicas and expose port 80 for HTTP traffic:

oc create deploy nginx --image=nginx --replicas=2 --port=80

Expose this deployment as a service:

oc expose deploy nginx

View the deployment and corresponding service:

oc get svc,deploy -l app=nginx

Sample output:

			
NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/nginx   ClusterIP   10.217.5.118   <none>        80/TCP    54s
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   2/2     2            2           60s

Now run a pod with cURL pre-installed and use it to send a cURL request to our NGINX deployment:

			
oc run curlpod --image=curlimages/curl -- sleep infinity
oc wait --for=condition=Ready pods --all --timeout=300s
oc exec curlpod -- curl -s nginx

Sample output of the last command:

			
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

		

Let’s go further and forward NGINX locally at port 8080 with port-forward:

oc port-forward svc/nginx 8080:80

In a new terminal window, make oc available for your session, then follow the NGINX logs to observe new requests coming in in the next step:

			
eval $(crc oc-env)
oc logs -f -l app=nginx

Now open a web browser and visit localhost:8080. Refresh the page 1 or 2 times.

Here’s what we get in our logs:

			
::1 - - [11/Jan/2024:12:54:05 +0000] "GET / HTTP/1.1" 200 615 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" "-"
2024/01/11 12:54:05 [error] 30#30: *7 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: ::1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "localhost:8080", referrer: "http://localhost:8080/"
::1 - - [11/Jan/2024:12:54:05 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://localhost:8080/" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" "-"
::1 - - [11/Jan/2024:12:54:13 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" "-"
::1 - - [11/Jan/2024:12:54:14 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" "-"
::1 - - [11/Jan/2024:12:55:04 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" "-"

		

A 200 OK response on the initial visit by our web browser (Firefox in my case) plus a failed attempt to fetch the favicon (since we don’t have one), then 304 Not Modified responses for the subsequent reloads which prevents NGINX from sending the contents of the same webpage over and over, improving response times and reducing the amount of network traffic.

Hopefully all of this is sufficient to convince you that OpenShift is in fact Kubernetes and oc is kubectl.

Let’s delete our NGINX deployment and associated resources – close the 2nd terminal window, return to the 1st one then press Ctrl+C to stop the port forwarding and run the following commands:

			
oc delete svc nginx
oc delete deploy nginx
oc delete po curlpod

Before we move on to the next section, let’s see how oc is more than just kubectl by inspecting a command that only exists with oc – feel free to run the command as well to view associated help information:

oc adm policy add-scc-to-user --help

Sample output:

			
Add a security context constraint to users or a service account.
Examples:
  # Add the 'restricted' security context constraint to user1 and user2
  oc adm policy add-scc-to-user restricted user1 user2
  # Add the 'privileged' security context constraint to serviceaccount1 in the
current namespace
  oc adm policy add-scc-to-user privileged -z serviceaccount1
Options:
    --allow-missing-template-keys=true:
    If true, ignore any errors in templates when a field or map key is
    missing in the template. Only applies to golang and jsonpath output
    formats.
    --dry-run='none':
    Must be "none", "server", or "client". If client strategy, only print
    the object that would be sent, without sending it. If server strategy,
    submit server-side request without persisting the resource.
    -o, --output='':
    Output format. One of: (json, yaml, name, go-template,
    go-template-file, template, templatefile, jsonpath, jsonpath-as-json,
    jsonpath-file).
    -z, --serviceaccount=[]:
    service account in the current namespace to use as a user
    --show-managed-fields=false:
    If true, keep the managedFields when printing objects in JSON or YAML
    format.
    --template='':
    Template string or path to template file to use when -o=go-template,
    -o=go-template-file. The template format is golang templates
    [http://golang.org/pkg/text/template/#pkg-overview].
Usage:
  oc adm policy add-scc-to-user SCC (USER | -z SERVICEACCOUNT) [USER ...]
[flags] [options]
Use "oc options" for a list of global command-line options (applies to all
commands).

		

This command add-scc-to-user is responsible for adding a security context constraint (SCC) to a given user or service account. SCCs, as its name suggests, is a form of mandatory access control (MAC) that places constraints on the security context that a Pod running as a particular user or service account can have, essentially SELinux but for OpenShift.

Exploring our cluster with the OpenShift web console

So far, we’ve only interacted with OpenShift via the command line, but what makes OpenShift really shine is its out-of-the-box integrated web console which helps in quickly visualizing the state of the entire cluster plus individual microservices, greatly easing the burden on developer and operations staff – no more 20 questions with kubectl 😉

Point your browser to console-openshift-console.apps-crc.testing and ignore the certificate warnings reported by your browser, which should bring you to the OpenShift login portal.

Now log in as a developer with developer as both the username and password, which should bring you to the Developer dashboard and greet you with a guided tour.

Check out the guided tour and feel free to explore the Developer perspective of the OpenShift web console before we proceed with the next section.

While still logged in as a developer, click the “Developer” dropdown to the top left and switch to the “Administrator” perspective.

OpenShift web admin dashboard from a developer's perspective

Notice how the administrator dashboard only prompts you to create a new project and doesn’t display any useful information about the cluster itself. For example, if you select “Workloads > Pods”, it’ll inform you that your access is restricted and you are not able to list all Pods running on the cluster. This is role-based access control (RBAC) in action for which OpenShift provides sane defaults for unprivileged users and greatly eases the management of RBAC compared to Kubernetes where users, groups, (cluster) roles and (cluster) role bindings must be created and defined manually.

OpenShift web admin pods - restricted view for developers

Now log out of the developer account and switch to the kubeadmin account which has cluster-wide administrator access, using the password you noted earlier. In case you forgot to take note of the password, you may retrieve it with crc as follows:

crc console --credentials

Sample output (password elided for security reasons):

			
To login as a regular user, run 'oc login -u developer -p developer https://api.crc.testing:6443'.
To login as an admin, run 'oc login -u kubeadmin -p XXXXX-XXXXX-XXXXX-XXXXX https://api.crc.testing:6443'

Once logged in as kubeadmin, notice how you are able to immediately view a summary of the entire cluster via the Administrator dashboard.

Again, feel free to poke around and get a feel of the Administrator perspective in OpenShift before we proceed to the next section.

Setting up Argo CD with the Operator

Another aspect of OpenShift that really shines is the extensive use of software operators to install and manage the lifecycle of cluster-level applications and add-on features, all through the Operator Hub available from the web console which greatly simplifies the operator installation process by making it only a few clicks away. Software operators encode human operational knowledge of managing a particular application into a software controller that performs these operational tasks in a fully automated manner, thus freeing the cluster administrator from the mundane task of managing that particular application and allowing them to focus on tasks that deliver actual business value.

While the operator pattern is not unique to OpenShift (see Canonical’s Juju charms for another example), the concept of Kubernetes operators originated from OpenShift and was donated by Red Hat to the CNCF as the Operator Framework which is now in the CNCF incubator and available to be installed on any standard Kubernetes cluster. The main value proposition that OpenShift offers in this regard is the native support for operators baked into OpenShift during cluster installation time and the ease of installing them via the web console as opposed to crafting the required custom resource definitions (CRDs) by hand and applying them via the command line.

In the following section, we will install the Argo CD operator on OpenShift via the web console and create an Argo CD cluster by defining a minimal ArgoCD custom resource, which will automate the installation and complete lifecycle management of the entire Argo CD instance in a fully transparent fashion. Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes which eases the implementation of modern application deployment patterns such as blue-green and canary deployments via Git-centric workflows, though we won’t focus on Argo CD itself for the purposes of this lab.

Next, search for “Argo CD” and select the application.

OpenShift will warn that this is a community Operator which is not officially supported by Red Hat. Acknowledge the warning by clicking “Continue”.

View the description for the Argo CD operator and click “Install”.

Before confirming the installation, OpenShift allows you to customize a few options such as selecting the version to install and the channel to install from. Leave the options at their defaults and click “Install” to proceed with the installation.

Now wait a few minutes while the operator is being installed until the page below indicates “Installed operator: ready for use” and you should see a green checkmark.

Congratulations – you have successfully installed your first operator on OpenShift!

Click the “View operator” button at the middle of the page to view the details of your newly installed Argo CD operator.

Notice the provided API ArgoCD by the Argo CD operator to the top left. This is a high-level CRD defined by the operator representing an entire Argo CD instance, so all we need to do is create a custom resource of type ArgoCD and apply it to the cluster, then the operator will automatically deploy an Argo CD instance and manage the entire lifecycle of that instance for us – no manual operation required. This also implies that we can create multiple Argo CD instances in different namespaces and have the operator manage all of them for us transparently, by simply defining multiple ArgoCD custom resources and applying them to our OpenShift cluster.

Open a new terminal window, make oc available in our PATH, then create the argocd namespace for our Argo CD instance and create a corresponding minimal ArgoCD custom resource by leaving the .spec field empty.

			
eval $(crc oc-env)
oc create ns argocd
oc apply -f - << EOF
apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: argocd
  namespace: argocd
spec: {}
EOF

		

Now wait up to 5 minutes for the Argo CD instance to become available:

			
oc -n argocd wait \
    --for=jsonpath='{.status.phase}'=Available \
    argocd \
    --all \
    --timeout=300s

		

Sample output:

argocd.argoproj.io/argocd condition met

Let’s see the pods created for our Argo CD instance:

oc -n argocd get po

Sample output:

			
NAME                                  READY   STATUS    RESTARTS   AGE
argocd-application-controller-0       1/1     Running   0          14m
argocd-redis-9c4df56bc-mqh8d          1/1     Running   0          14m
argocd-repo-server-689fdf7d9d-ngpn8   1/1     Running   0          14m
argocd-server-76c4849577-j7l5n        1/1     Running   0          14m

		

The Argo CD web console is available as the argocd-server Service which is a ClusterIP by default and therefore only accessible from within the cluster. Forward this service locally to port 8443, then open localhost:8443 in a new browser window and ignore any certificate warnings reported by your browser which presents us with a login portal.

oc -n argocd port-forward svc/argocd-server 8443:443

The login username is admin and the initial login password is stored in the secret argocd-cluster within the key admin.password.

Open a new terminal window (again) and make oc available in our PATH.

eval $(crc oc-env)

Now fetch the initial login password, first by using the kubectl way of doing things.

			
oc -n argocd get secret \
    argocd-cluster \
    -o jsonpath='{.data.admin\.password}' | \
    base64 -d -

Notice how this command is difficult to read and understand since it doesn’t tell us what we want to achieve (extract the initial login password); instead, it only tells us how to achieve it (get the .data.admin\.password field in the argocd-cluster secret and decode it as Base64).

Fortunately, oc provides a more readable alternative to extracting the initial login password from our secret:

oc -n argocd extract secret/argocd-cluster --to=-

Finally, log in to the Argo CD web console with the username admin and the password as output by the command above which should bring us to the Argo CD dashboard.

Cleaning up

Tearing down our OKD cluster with CRC is simple – simply run:

crc delete

Answer y when prompted.

Concluding remarks and going further

OpenShift is a full-fledged IDP developed by Red Hat on top of Kubernetes which greatly enhances the developer experience, thereby improving developer productivity, reducing time to market and enabling enterprises to focus on delivering business value instead of performing the undifferentiated heavy lifting of building and maintaining their own IDP. Throughout the lab, we saw 2 major value-added features of OpenShift which makes all of this possible, namely the integrated OpenShift web console and the ability to install software controllers known as operators with a few clicks which manage the entire lifecycle of applications on behalf of the cluster administrator.

This article is by no means a comprehensive introduction to OpenShift. If you would like to dive into OpenShift, consider the following resources:

Introduction to Containers, Kubernetes and OpenShift, a self-paced online course by IBM on edX available at no cost (Red Hat is a subsidiary of IBM)
The Red Hat OpenShift skill paths recommended by Red Hat

Additionally, if you would like to dive into Argo CD, consider the following resources:

Read through the Argo CD documentation and try out some of the tutorials there
The Certified Argo Project Associate (CAPA) exam and associated training material offered by the Linux Foundation

I hope you enjoyed this article and stay tuned for new content 😉

+ GitOps, Kubernetes

Running stateful workloads on Kubernetes with Rook Ceph

2023-12-26

+

+

+

+

+

+
The source code for this lab exercise is available on GitHub.

In our last article Investigating a failed VolumeSnapshot with NFS on Kubernetes, we saw how using NFS as a storage backend for stateful workloads on Kubernetes is not suitable in a production context due to fundamental limitations of NFS itself. So how are we to run our stateful applications on Kubernetes, if at all?

A common deployment model for running stateful applications on Kubernetes is the cloud native hybrid architecture where stateful components of an application such as a database or object storage run on virtual machines (VMs) for optimal stability and performance, while stateless components such as the frontend web UI or REST API server run on Kubernetes for maximal resiliency, elasticity and high availability. While this deployment model combines the best of both worlds, configuration is complex compared to deploying the entire application in-cluster since the stateless components running on Kubernetes must be explicitly configured to point to the out-of-cluster stateful components instead of leveraging the native service discovery mechanisms offered by Kubernetes.

Another option is to leverage a Kubernetes-native distributed storage solution such as Rook Ceph as the storage backend for stateful components running on Kubernetes. This has the benefit of simplifying application configuration while addressing business requirements for data backup and recovery such as the ability to take volume snapshots at a regular interval and perform application-level data recovery in case of a disaster.

Rook Ceph is a CNCF Graduated project. As its name might suggest, Rook Ceph is comprised of two major components: Ceph and Rook. Ceph is the distributed storage system itself while Rook is the Kubernetes operator for automated setup and lifecycle management of Ceph clusters which greatly simplifies the deployment and administration of Ceph clusters on Kubernetes.

In the lab to follow, we’ll quickly provision a 3-node kubeadm cluster (1 master, 2 workers) on the cloud provider of your choice using an automation stack comprised of OpenTofu and Ansible, then deploy Rook Ceph using the official Helm charts and confirm that we are now able to successfully create CSI volume snapshots from PVCs by reusing the MinIO example from our last article.

Lab: Deploying Rook Ceph to a 3-node kubeadm cluster

This lab has been tested with Kubernetes v1.29.0 (Mandala).

Prerequisites

Familiarity with Kubernetes cluster administration is assumed. Furthermore, it is assumed that you are already familiar with Kubernetes CSI and related API objects such as VolumeSnapshots and VolumeSnapshotClasses. If not, it is recommended to follow through the lab at Investigating a failed VolumeSnapshot with NFS on Kubernetes before attempting this lab.

Setting up your environment

A Unix/Linux environment is required. If on Windows, make sure WSL2 is installed and follow the lab within WSL2.

The automated 3-node kubeadm cluster setup is comprised of 2 components: OpenTofu and Ansible. OpenTofu is an open source drop-in replacement for the infamous Terraform infrastructure-as-code (IaC) tool governed by the Linux Foundation, while Ansible is an automated scripting and configuration management tool. We’ll use OpenTofu to provision the necessary infrastructure on the cloud provider of your choice (currently AWS and Alibaba Cloud are supported), then switch over to Ansible for running the scripts required to set up a bare-bones Kubernetes cluster on our cloud infrastructure.

In case you do not have an AWS or Alibaba Cloud account and do not wish to create one, it is still possible to follow through this lab by manually provisioning the required infrastructure on the cloud provider of your choice, on-premises as VMs or on bare metal:
- 3 nodes running Ubuntu 22.04 LTS
- At least 8 vCPUs per node
- At least 32 GiB memory per node
- At least 16 GiB for the system disk per node
- At least 1 unpartitioned, unformatted 64 GiB data disk per node
- SSH with public key authentication must be enabled for all nodes (or customize the Ansible playbook accordingly)
- All nodes must have an account with the same username and SSH key (or customize the Ansible playbook accordingly)
- Passwordless sudo must be configured (or customize the Ansible playbook accordingly)
You are then free to skip the step involving OpenTofu and invoke ansible-playbook directly, after customizing the 2 files detailed below with configuration values matching your own infrastructure.

ansible/ansible.cfg
[defaults] inventory = ./hosts.yaml remote_user = ubuntu private_key_file = /path/to/your/key.pem host_key_checking = False
ansible/hosts.yaml
masters: hosts: master0: ansible_host: x.x.x.x private_ip: x.x.x.x workers: hosts: worker0: ansible_host: x.x.x.x worker1: ansible_host: x.x.x.x
The rest of the instructions will assume an AWS environment, though note that Alibaba Cloud is also supported with minimal modification, namely by setting the following environment variables:
export CLOUD_PROVIDER="aliyun" export TF_VAR_aliyun_access_key="XXXXXXXXXXXXXXXXXXXXXXXX" # replace me! export TF_VAR_aliyun_secret_key="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" # replace me!
Setting up your AWS account

You’ll need to set up your AWS account and create an IAM administrator, then generate an access key and secret key for your IAM administrator for setting up AWS CLI v2 in our next step. Consult the official AWS documentation if in doubt.

Setting up AWS CLI v2

First make sure ~/.local/bin/ exists and is in your PATH so sudo is not required for installing the various command-line tools in this lab on your local laptop / desktop / workstation.
mkdir -p "$HOME/.local/bin/" echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> "$HOME/.bashrc" source "$HOME/.bashrc"
Now download AWS CLI v2 from the official website and install it using the provided script:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip -n awscliv2.zip ./aws/install --bin-dir "$HOME/.local/bin/" --install-dir "$HOME/.local/aws-cli/"
You’ll also need to configure AWS CLI v2 with your access and secret keys when prompted:
aws configure
Confirm that the setup is functional:
aws ec2 describe-instances
Sample output:
{ "Reservations": [] }
Installing OpenTofu

Now install OpenTofu using the official release binaries. The latest version is 1.6.0-rc1 at the time of writing.
wget https://github.com/opentofu/opentofu/releases/download/v1.6.0-rc1/tofu_1.6.0-rc1_linux_amd64.zip unzip -n tofu_1.6.0-rc1_linux_amd64.zip mv tofu "$HOME/.local/bin/"
Confirm that OpenTofu is correctly installed:
tofu -version
Sample output:
OpenTofu v1.6.0-rc1 on linux_amd64
Installing Ansible

The recommended installation method is via pipx which needs to be first installed.
python3 -m pip install --user pipx python3 -m pipx ensurepath
Now install Ansible with pipx. The latest version is 2.16.2 at the time of writing.
python3 -m pipx install --include-deps ansible
Confirm that Ansible is correctly installed:
ansible-playbook --version
Sample output:
ansible-playbook [core 2.16.2] config file = None configured module search path = ['/home/donaldsebleung/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /home/donaldsebleung/.local/share/pipx/venvs/ansible/lib/python3.10/site-packages/ansible ansible collection location = /home/donaldsebleung/.ansible/collections:/usr/share/ansible/collections executable location = /home/donaldsebleung/.local/bin/ansible-playbook python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/home/donaldsebleung/.local/share/pipx/venvs/ansible/bin/python) jinja version = 3.1.2 libyaml = True
Deploying the automation stack

Clone the project repository and make it your working directory.
git clone https://github.com/DonaldKellett/kubeadm-1m2w.git cd kubeadm-1m2w/
Now invoke OpenTofu to provision the necessary Amazon EC2 instances. Note that Rook Ceph has nontrivial resource requirements so 8 vCPUs and 32 GiB of memory per node are required, which is satisfied by the t3.2xlarge instance type.
export CLOUD_PROVIDER="aws" export TF_VAR_instance_type="t3.2xlarge" tofu -chdir="opentofu/${CLOUD_PROVIDER}/" init tofu -chdir="opentofu/${CLOUD_PROVIDER}/" apply
Answer yes when prompted. Once the provisioning is complete, make note of the k8s-master0-public-ip – you’ll need this to SSH into the master node for running kubectl commands.

Now wait a few seconds for the instances to stabilize and run the provided Ansible playbook which should take no longer than 5 minutes to complete:
export ANSIBLE_CONFIG="${PWD}/ansible/ansible.cfg" ansible-playbook "${PWD}/ansible/playbook.yaml"
Checking that everything is set up correctly

Now SSH into the master node using the public IP you noted earlier and check that all 3 nodes are up and ready:
kubectl get nodes
Sample output:
NAME STATUS ROLES AGE VERSION master0 Ready control-plane 3m43s v1.29.0 worker0 Ready <none> 102s v1.29.0 worker1 Ready <none> 102s v1.29.0
Let’s also take a look at the disks available to the master node (also available to each worker node):
sudo lsblk -f
Sample output:
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS loop0 0 100% /snap/amazon-ssm-agent/7628 loop1 0 100% /snap/core18/2812 loop2 0 100% /snap/core20/2015 loop3 0 100% /snap/lxd/24322 loop4 0 100% /snap/snapd/20290 nvme1n1 nvme0n1 ├─nvme0n1p1 ext4 1.0 cloudimg-rootfs 9e71e708-e903-4c26-8506-d85b84605ba0 11G 28% / ├─nvme0n1p14 └─nvme0n1p15 vfat FAT32 UEFI A62D-E731 98.3M 6% /boot/efi
Notice that the 64 GiB data disk nvme1n1 is unformatted and unpartitioned. This is important as our Ceph cluster will use this data disk later (on each node) for providing storage to our stateful Kubernetes applications.

All subsequent commands in this lab should be executed on the Kubernetes master node unless otherwise instructed.

Installing Rook Ceph

Before installing Rook Ceph, we’ll need to install a few dependencies:
- The VolumeSnapshot, VolumeSnapshotContent and VolumeSnapshotClass CRDs
- Helm
- The CSI snapshot controller
Install the CRDs:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
Confirm the CRDs are correctly installed:
kubectl api-resources | grep volumesnapshot
Sample output:
volumesnapshotclasses vsclass,vsclasses snapshot.storage.k8s.io/v1 false VolumeSnapshotClass volumesnapshotcontents vsc,vscs snapshot.storage.k8s.io/v1 false VolumeSnapshotContent volumesnapshots vs snapshot.storage.k8s.io/v1 true VolumeSnapshot
Now ensure ~/.local/bin/ is in your PATH so sudo is not required for installing Helm:
mkdir -p "$HOME/.local/bin/" echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> "$HOME/.bashrc" source "$HOME/.bashrc"
Next, install Helm:
wget https://get.helm.sh/helm-v3.13.3-linux-amd64.tar.gz tar xvf helm-v3.13.3-linux-amd64.tar.gz mv linux-amd64/helm "$HOME/.local/bin/"
Confirm Helm is correctly installed:
helm version
Sample output:
version.BuildInfo{Version:"v3.13.3", GitCommit:"c8b948945e52abba22ff885446a1486cb5fd3474", GitTreeState:"clean", GoVersion:"go1.20.11"}
Now we can install the snapshot controller via a Helm chart using the default values.
helm repo add democratic-csi https://democratic-csi.github.io/charts/ helm repo update helm -n kube-system install \ snapshot-controller \ democratic-csi/snapshot-controller \ --version 0.2.4
Installing the Rook operator

The Rook operator performs lifecycle management functions of Ceph clusters and must be installed prior to the Ceph cluster itself. By default, it expects to be deployed in the namespace rook-ceph, though this behavior is configurable.

Remember to specify the option enableDiscoveryDaemon=true which automatically scans each node on a regular interval for clean (unformatted and unpartitioned), newly attached data disks to be added to the storage pool.
helm repo add rook-release https://charts.rook.io/release helm repo update helm -n rook-ceph install \ rook-ceph \ rook-release/rook-ceph \ --set enableDiscoveryDaemon=true \ --create-namespace
Now wait for the operator and per-node discovery daemon to enter a Ready state:
kubectl -n rook-ceph wait \ --for=condition=Ready \ pods \ -l 'app in (rook-ceph-operator, rook-discover)' \ --timeout=180s
Sample output:
pod/rook-ceph-operator-775858d6b-98qsk condition met pod/rook-discover-ln4gp condition met pod/rook-discover-tgzdn condition met pod/rook-discover-v779r condition met
If you only see the first line of output, re-run the command above since the discovery daemon pods are not created until the operator is up and running.

Installing the Ceph cluster

Now install the Ceph cluster – we’ll briefly go through each of the key components in a moment.

Ensure that the following Helm chart values are set:
- cephBlockPoolsVolumeSnapshotClass.enabled=true for generating the RBD VolumeSnapshotClass automatically
- cephFileSystemVolumeSnapshotClass.enabled=true for generating the Ceph filesystem VolumeSnapshotClass automatically
helm -n rook-ceph install \ rook-ceph-cluster \ rook-release/rook-ceph-cluster \ --set cephBlockPoolsVolumeSnapshotClass.enabled=true \ --set cephFileSystemVolumeSnapshotClass.enabled=true
Now wait for the Ceph cluster to enter a Ready state – this may take up to 15 minutes:
kubectl -n rook-ceph wait --for=condition=Ready cephclusters --all --timeout=900s
Sample output:
cephcluster.ceph.rook.io/rook-ceph condition met
Let’s take a closer look at our Ceph cluster.
kubectl -n rook-ceph get cephcluster rook-ceph
Sample output:
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL FSID rook-ceph /var/lib/rook 3 45m Ready Cluster created successfully HEALTH_OK 5d6cc90c-87f1-4de7-8b6d-8c18562a9fa0
HEALTH_OK indicates that our Ceph cluster is healthy. To really see what’s going on, it may help to view the created pods.
kubectl -n rook-ceph get pods -l 'app,app notin (rook-ceph-operator, rook-discover)'
Sample output:
NAME READY STATUS RESTARTS AGE csi-cephfsplugin-4jrkb 2/2 Running 1 (46m ago) 47m csi-cephfsplugin-provisioner-cd88d8c4-fcm9g 5/5 Running 1 (46m ago) 47m csi-cephfsplugin-provisioner-cd88d8c4-jbwcb 5/5 Running 0 47m csi-cephfsplugin-pzbgx 2/2 Running 0 47m csi-cephfsplugin-w6f6h 2/2 Running 1 (46m ago) 47m csi-rbdplugin-gbrc8 2/2 Running 1 (46m ago) 47m csi-rbdplugin-kl6cd 2/2 Running 1 (46m ago) 47m csi-rbdplugin-provisioner-f6d4c9775-wlb9n 5/5 Running 1 (46m ago) 47m csi-rbdplugin-provisioner-f6d4c9775-xvvwq 5/5 Running 0 47m csi-rbdplugin-s89gt 2/2 Running 0 47m rook-ceph-crashcollector-master0-5787b75899-xh62s 1/1 Running 0 41m rook-ceph-crashcollector-worker0-6d859cbd7-tpcvm 1/1 Running 0 42m rook-ceph-crashcollector-worker1-6f8d676454-hrlfx 1/1 Running 0 37m rook-ceph-mds-ceph-filesystem-a-5977dbcfc8-m6tp7 2/2 Running 0 42m rook-ceph-mds-ceph-filesystem-b-57d57cc6d4-v542c 2/2 Running 0 41m rook-ceph-mgr-a-6d48d5cd4-l5jq5 3/3 Running 0 45m rook-ceph-mgr-b-7b944ddd97-skknv 3/3 Running 0 45m rook-ceph-mon-a-764cbbbd9f-hdmtx 2/2 Running 0 47m rook-ceph-mon-b-5b6cd88d59-mh4p4 2/2 Running 0 45m rook-ceph-mon-c-795c785489-p65nr 2/2 Running 0 45m rook-ceph-osd-0-bb954988-86rx6 2/2 Running 0 44m rook-ceph-osd-1-766d6f9dd5-ct4np 2/2 Running 0 44m rook-ceph-osd-2-6f9df74867-ntjjg 2/2 Running 0 44m rook-ceph-osd-prepare-master0-4gpbg 0/1 Completed 0 43m rook-ceph-osd-prepare-worker0-zkvdm 0/1 Completed 0 43m rook-ceph-osd-prepare-worker1-nlq89 0/1 Completed 0 43m rook-ceph-rgw-ceph-objectstore-a-86cc9554df-476vf 2/2 Running 0 37m
That’s a lot of pods! We’ll focus on the important ones and skip the rest:
- Ceph monitor (MON): responsible for maintaining the state of the cluster. Like etcd, it relies on a quorum so 3 MONs are required for high availability. On Kubernetes, they run as Pods rook-ceph-mon-{a,b,c}*, one on each node
- Ceph manager (MGR): responsible for interacting with external monitoring and management systems. Runs in active-standby mode so 2 replicas are sufficient. On Kubernetes, they run as Pods rook-ceph-mgr-{a,b}*
- Ceph OSD: daemon for interacting with logical volumes, one per volume. Stands for “Object Storage Daemon”. On Kubernetes, they run as Pods rook-ceph-osd-n* where n is a non-negative integer
- Ceph block device (RBD): also known as RADOS block device (hence the abbreviation RBD). Provides block-level storage to stateful Kubernetes applications
Now that our Ceph cluster is running, let’s reuse the MinIO example from our last article and confirm that we are now able to successfully create a VolumeSnapshot from a PVC. But before that, let’s confirm that we have a working StorageClass for creating PVCs and a working VolumeSnapshotClass for creating volume snapshots from our PVCs.
kubectl get storageclass kubectl get volumesnapshotclass
Sample output:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE ceph-block (default) rook-ceph.rbd.csi.ceph.com Delete Immediate true 66m ceph-bucket rook-ceph.ceph.rook.io/bucket Delete Immediate false 66m ceph-filesystem rook-ceph.cephfs.csi.ceph.com Delete Immediate true 66m NAME DRIVER DELETIONPOLICY AGE ceph-block rook-ceph.rbd.csi.ceph.com Delete 66m ceph-filesystem rook-ceph.cephfs.csi.ceph.com Delete 66m
Observe that RBD is marked as the default StorageClass and has a corresponding VolumeSnapshotClass so we expect to be able to create snapshots from our PVCs normally.

Installing MinIO and creating a VolumeSnapshot

This is much the same as with our previous lab.

Add the Bitnami repo and install MinIO:
helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update helm -n minio install \ minio \ bitnami/minio \ --version 12.9.4 \ --create-namespace
Now wait for it to become ready, which should take no longer than 5 minutes:
kubectl -n minio wait --for=condition=Ready pods --all --timeout=300s
Observe that a PVC minio is created and bound:
kubectl -n minio get pvc minio
Now create a snapshot from our MinIO PVC:
kubectl -n minio apply -f - << EOF --- apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: minio spec: volumeSnapshotClassName: ceph-block source: persistentVolumeClaimName: minio EOF
Wait for our snapshot to become ready:
kubectl -n minio wait \ --for=jsonpath='{.status.readyToUse}'=true \ volumesnapshot \ minio \ --timeout=180s
Sample output:
volumesnapshot.snapshot.storage.k8s.io/minio condition met
Cleaning up

Unless you wish to continue experimenting with your cluster, log out of the master node now, ensure that your working directory is set to the kubeadm-1m2w/ project you cloned earlier and run the following commands on your own laptop / desktop / workstation to clean up the lab resources to save costs:
export CLOUD_PROVIDER="aws" tofu -chdir="opentofu/${CLOUD_PROVIDER}/" destroy
Answer yes when prompted.

Concluding remarks and going further

By deploying the production-grade Rook Ceph CSI distributed storage backend to Kubernetes, it is possible to run stateful applications directly on Kubernetes in a production context and combine it with integrated backup and disaster recovery (DR) solutions capable of cross-cluster application-level data recovery such as Velero and Kasten K10 to meet your business continuity, legal, audit and compliance requirements, all without sacrificing the ability to leverage Kubernetes’ powerful features and abstractions such as built-in service discovery mechanisms for managing your entire application. However, in rare cases where performance, stability and permanence* are of utmost importance, it may still make sense to deploy specific stateful component(s) outside of Kubernetes and adopt a partial cloud native hybrid model, just for those affected components.

Storage (CSI) on Kubernetes has always been a tricky subject and has taken a long time to mature compared to other fundamental capabilities such as compute (CRI) and networking (CNI), leaving IT practitioners and managers the impression that “Kubernetes is best for running stateless workloads, run stateful workloads in VMs”. However, as Kubernetes’ storage capabilities mature over time, this fundamental notion is under challenge. So, what do you think?

* From the GitLab 3k reference architecture with a cloud native hybrid deployment model:

Hybrid installations leverage the benefits of both cloud native and traditional compute deployments. With this, stateless components can benefit from cloud native workload management benefits while stateful components are deployed in compute VMs with Linux package installations to benefit from increased permanence.
+

+

+

+

+

+

+ Cloud, Kubernetes, Storage

+

+

+

+

+

+

+

+

+

+

+

+

Investigating a failed VolumeSnapshot with NFS on Kubernetes

2023-11-14

Based on real-world experience.

Two common mistakes of using NFS for dynamic volume provisioning on Kubernetes:

Using nfs-subdir-external-provisioner instead of csi-driver-nfs
Using NFS for dynamic volume provisioning on Kubernetes

Using `nfs-subdir-external-provisioner` instead of `csi-driver-nfs`

The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. Using CSI third-party storage providers can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code.

Source: Kubernetes CSI Developer Documentation

Basically, CSI is a unified abstraction layer for managing and interacting with storage on Kubernetes. A conforming implementation is known as a CSI driver and exposes the following common storage operations which covers both the running of stateful applications on Kubernetes and primitives for enabling application backup and recovery:

The ability to define a StorageClass utilizing the driver which dynamically provisions PVs to satisfy PVC requests
The ability to define a VolumeSnapshotClass utilizing the driver which dynamically creates volume snapshots (VolumeSnapshotContent) to satisfy VolumeSnapshot requests

csi-driver-nfs is a CSI driver for NFS. On the other hand, nfs-subdir-external-provisioner is just a StorageClass implementation and lacks volume snapshot capabilities – not acceptable for production use.

Using NFS for dynamic volume provisioning on Kubernetes

In some cases, volume snapshots fail to be created even with the NFS CSI driver. This is due to fundamental limitations with NFS itself. Perhaps a hands-on demo is in order.

In the lab to follow, we’ll set up a 2-node kubeadm cluster (1 master, 1 worker) with the NFS CSI driver installed, deploy MinIO as a sample stateful application and attempt to create a VolumeSnapshot from the minio PVC. We’ll then investigate the root cause of the issue and conclude why NFS is not suitable for Kubernetes storage in a production context.

Lab: NFS VolumeSnapshots failing in action

Prerequisites

Familiarity with Kubernetes cluster administration is assumed. If not, consider enrolling in the comprehensive LFS258: Kubernetes Fundamentals online training course offered by The Linux Foundation which is also the official training course for the CKA certification exam offered by the CNCF.

Setting up your environment

It is assumed you already have a public cloud account such as an AWS account or a laptop / workstation capable of hosting at least 2 Linux nodes each with 2 vCPUs, 8G of RAM and 40G of storage space, one of which will become the master node and the other the worker node. You may follow the lab with a bare-metal setup as well if desired.

The reference distribution is Ubuntu 22.04 LTS for which the instructions in this lab have been tested against. If you’re looking for a challenge, feel free to follow the lab with a different distribution but beware that some of the instructions may require non-trivial modification.

For the purposes of this lab, we’ll refer to our master node as master0 and worker node as worker0.

Provisioning a two-node kubeadm cluster

Let’s set up a kubeadm cluster following the typical process.

The versions of Kubernetes and associated components to be installed:

Kubernetes 1.28.3
containerd 1.7.8
runc 1.1.10
CNI plugins 1.3.0
Calico 3.26.3

Setting up `master0`

Run the following commands on master0 to perform preliminary setup and avoid issues on installing and initializing Kubernetes. Make sure to replace x.x.x.x below with the private IP address of master0.

			
sudo hostnamectl set-hostname master0
echo "export PATH=\"/opt/cni/bin:/usr/local/sbin:/usr/local/bin:\$PATH\"" >> "$HOME/.bashrc" && \
    source "$HOME/.bashrc"
sudo sed -i 's#Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin#Defaults    secure_path = /opt/cni/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin#' /etc/sudoers
export K8S_CONTROL_PLANE="x.x.x.x"
echo "$K8S_CONTROL_PLANE k8s-control-plane" | sudo tee -a /etc/hosts
sudo modprobe br_netfilter
echo br_netfilter | sudo tee /etc/modules-load.d/kubernetes.conf
cat << EOF | sudo tee -a /etc/sysctl.conf
net.ipv4.ip_forward=1
EOF
sudo sysctl -p
sudo systemctl reboot

		

After the reboot, run the following commands to install the containerd CRI and associated components:

			
wget https://github.com/containerd/containerd/releases/download/v1.7.8/containerd-1.7.8-linux-amd64.tar.gz
sudo tar Cxzvf /usr/local containerd-1.7.8-linux-amd64.tar.gz
sudo mkdir -p /usr/local/lib/systemd/system/
sudo wget -qO /usr/local/lib/systemd/system/containerd.service https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo systemctl daemon-reload
sudo systemctl enable --now containerd.service
sudo mkdir -p /etc/containerd/
containerd config default | \
    sed 's/SystemdCgroup = false/SystemdCgroup = true/' | \
    sed 's/pause:3.8/pause:3.9/' | \
    sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd.service
sudo mkdir -p /usr/local/sbin/
sudo wget -qO /usr/local/sbin/runc https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64
sudo chmod +x /usr/local/sbin/runc
sudo mkdir -p /opt/cni/bin/
wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz

		

Now run the commands below to install Kubernetes and initialize our master node:

			
sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update && sudo apt install -y \
    kubeadm=1.28.3-1.1 \
    kubelet=1.28.3-1.1 \
    kubectl=1.28.3-1.1
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet.service
cat > kubeadm-config.yaml << EOF
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.28.3
controlPlaneEndpoint: "k8s-control-plane:6443"
networking:
  podSubnet: "192.168.0.0/16"
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
EOF
sudo kubeadm init --config kubeadm-config.yaml
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
echo "source <(kubectl completion bash)" >> "$HOME/.bashrc" && \
    source "$HOME/.bashrc"
wget -qO - https://raw.githubusercontent.com/projectcalico/calico/v3.26.3/manifests/calico.yaml | \
    kubectl apply -f -

		

Now run the following command to wait for master0 to become ready – this should take no longer than 5 minutes:

kubectl wait --for=condition=Ready node master0

Sample output:

node/master0 condition met

Setting up `worker0`

Now set up our worker node worker0.

Again, the preliminary setup which also reboots our node – replace x.x.x.x again with the private IP address of master0:

			
sudo hostnamectl set-hostname worker0
echo "export PATH=\"/opt/cni/bin:/usr/local/sbin:/usr/local/bin:\$PATH\"" >> "$HOME/.bashrc" && \
    source "$HOME/.bashrc"
sudo sed -i 's#Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin#Defaults    secure_path = /opt/cni/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin#' /etc/sudoers
export K8S_CONTROL_PLANE="x.x.x.x"
echo "$K8S_CONTROL_PLANE k8s-control-plane" | sudo tee -a /etc/hosts
sudo modprobe br_netfilter
echo br_netfilter | sudo tee /etc/modules-load.d/kubernetes.conf
cat << EOF | sudo tee -a /etc/sysctl.conf
net.ipv4.ip_forward=1
EOF
sudo sysctl -p
sudo systemctl reboot

		

Next, install containerd and associated components:

			
wget https://github.com/containerd/containerd/releases/download/v1.7.8/containerd-1.7.8-linux-amd64.tar.gz
sudo tar Cxzvf /usr/local containerd-1.7.8-linux-amd64.tar.gz
sudo mkdir -p /usr/local/lib/systemd/system/
sudo wget -qO /usr/local/lib/systemd/system/containerd.service https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo systemctl daemon-reload
sudo systemctl enable --now containerd.service
sudo mkdir -p /etc/containerd/
containerd config default | \
    sed 's/SystemdCgroup = false/SystemdCgroup = true/' | \
    sed 's/pause:3.8/pause:3.9/' | \
    sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd.service
sudo mkdir -p /usr/local/sbin/
sudo wget -qO /usr/local/sbin/runc https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64
sudo chmod +x /usr/local/sbin/runc
sudo mkdir -p /opt/cni/bin/
wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz

		

Now, install Kubernetes and initialize our worker node – replace the x’s with your Kubernetes token and CA certificate hash as shown in the output of kubeadm init on our master node master0:

			
sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update && sudo apt install -y \
    kubeadm=1.28.3-1.1 \
    kubelet=1.28.3-1.1
sudo apt-mark hold kubelet kubeadm
sudo systemctl enable --now kubelet.service
export K8S_TOKEN="xxxxxx.xxxxxxxxxxxxxxxx"
export K8S_CA_CERT_HASH="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
sudo kubeadm join k8s-control-plane:6443 \
    --discovery-token "${K8S_TOKEN}" \
    --discovery-token-ca-cert-hash "sha256:${K8S_CA_CERT_HASH}"

		

Now run a similar command on the master node to wait for our worker node to become ready – again, this should take no longer than 5 minutes:

kubectl wait --for=condition=Ready node worker0

Sample output:

node/worker0 condition met

Setting up NFS and installing the NFS CSI driver

We’ll set up our NFS share on our worker node so our applications run close to our data.

Run these commands on worker0:

			
sudo apt update && sudo apt install -y nfs-kernel-server
sudo systemctl enable --now nfs-kernel-server.service
sudo chown -R nobody:nogroup /srv
cat << EOF | sudo tee /etc/exports
/srv     *(rw,sync)
EOF
sudo exportfs -a

		

Now check that the NFS share is available:

showmount -e

Sample output:

			
Export list for worker0:
/srv *

With our NFS share available, let’s install the NFS CSI driver onto our cluster. We’ll also create the following objects:

A StorageClass utilizing the driver for dynamically provisioning NFS-backed storage, marked as default via the annotation storageclass.kubernetes.io/is-default-class=true
A VolumeSnapshotClass utilizing the driver for taking snapshots of our NFS volumes

Run these commands on master0. Replace x.x.x.x below with the IP address of worker0 since that is where we installed our NFS share.

			
wget https://get.helm.sh/helm-v3.13.2-linux-amd64.tar.gz
tar xvf helm-v3.13.2-linux-amd64.tar.gz
chmod +x linux-amd64/helm
mkdir -p "$HOME/.local/bin/"
mv linux-amd64/helm "$HOME/.local/bin/"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> "$HOME/.bashrc"
source "$HOME/.bashrc"
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm repo update
helm -n kube-system install \
    csi-driver-nfs \
    csi-driver-nfs/csi-driver-nfs \
    --version v4.5.0 \
    --set externalSnapshotter.enabled=true
export K8S_WORKER_NODE="x.x.x.x"
kubectl apply -f - << EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: ${K8S_WORKER_NODE}
  share: /srv
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF
kubectl annotate storageclass nfs-csi storageclass.kubernetes.io/is-default-class=true
kubectl apply -f - << EOF
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-nfs-snapclass
driver: nfs.csi.k8s.io
deletionPolicy: Delete
EOF

		

Install MinIO for our stateful application

MinIO is an Amazon S3-compatible object storage solution that can be deployed on Kubernetes.

We’ll not dive deep into MinIO, however – our focus is on the fact that it is stateful and therefore requests provisioned storage via a PVC with the name minio.

Run the commands below on master0:

			
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm -n minio install \
    minio \
    bitnami/minio \
    --version 12.9.4 \
    --create-namespace

		

Now wait for the pods in minio namespace to become ready – this should take no longer than 5 minutes:

kubectl -n minio wait --for=condition=Ready pods --all

Sample output:

pod/minio-76c7dcbb5-84qmb condition met

Notice that a PVC minio has been created and its status should be Bound:

kubectl -n minio get pvc

Sample output:

			
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
minio   Bound    pvc-44ac176e-f1d5-4fe1-ac58-2153f0db6198   8Gi        RWO            nfs-csi        2m56s

Attempt to create a snapshot from the `minio` PVC

While creating a snapshot on its own does not constitute a backup, it is nonetheless a fundamental operation implemented as part of a complete workflow in comprehensive Kubernetes-native backup and recovery solutions such as Velero and Kasten K10.

Let’s try to create a VolumeSnapshot for our minio PVC – run these on master0:

			
kubectl -n minio apply -f - << EOF
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: test-minio-snapshot
spec:
  volumeSnapshotClassName: csi-nfs-snapclass
  source:
    persistentVolumeClaimName: minio
EOF

		

Now wait a few seconds and observe that the snapshot has failed:

			
kubectl -n minio get volumesnapshot test-minio-snapshot -o jsonpath='{.status.error.message}'

Sample output:

			
Failed to check and update snapshot content: failed to take snapshot of the volume 10.1.0.162#srv#pvc-44ac176e-f1d5-4fe1-ac58-2153f0db6198##: "rpc error: code = Internal desc = failed to create archive for snapshot: exit status 2: ./\n./.minio.sys/\n./.minio.sys/multipart/\n./.minio.sys/format.json\n./.minio.sys/pool.bin/\n./.minio.sys/pool.bin/xl.meta\n./.minio.sys/tmp/\n./.minio.sys/tmp/9bd8ec0e-cf8a-4c86-8dcf-0bbe1d35de80\n./.minio.sys/tmp/.trash/\n./.minio.sys/config/\n./.minio.sys/config/iam/\n./.minio.sys/config/iam/format.json/\n./.minio.sys/config/iam/format.json/xl.meta\n./.minio.sys/config/config.json/\n./.minio.sys/config/config.json/xl.meta\n./.minio.sys/buckets/\n./.minio.sys/buckets/.bloomcycle.bin/\n./.minio.sys/buckets/.bloomcycle.bin/xl.meta\n./.minio.sys/buckets/.usage.json/\n./.minio.sys/buckets/.usage.json/xl.meta\ntar: ./.root_password: Cannot open: Permission denied\ntar: ./.root_user: Cannot open: Permission denied\ntar: Exiting with failure status due to previous errors\n"

In particular, notice the keywords Permission denied.

Uncovering the root cause of the failed snapshot

Viewing the file and directory ownership under /bitnami/minio/data/ in the MinIO pod reveals the root cause of the issue:

			
export MINIO_POD="$(kubectl -n minio get pod -l app.kubernetes.io/name=minio --no-headers | awk '{ print $1 }')"
kubectl -n minio exec "${MINIO_POD}" -- ls -al /bitnami/minio/data/

Sample output:

			
total 20
drwxrwsr-x 3 nobody nogroup 4096 Nov 14 14:02 .
drwxr-xr-x 3 root   root    4096 Nov 11 18:58 ..
drwxr-sr-x 7   1001 nogroup 4096 Nov 14 14:02 .minio.sys
-rw------- 1   1001 nogroup   11 Nov 14 14:02 .root_password
-rw------- 1   1001 nogroup    6 Nov 14 14:02 .root_user

		

By default, NFS shares have root squash enabled which maps the privileged root user on NFS clients to the unprivileged nobody user on the NFS server, preventing remote clients from unexpectedly gaining root privileges on the NFS host which is a security concern. All other users are mapped to their own UID.

The MinIO pod in our Helm chart runs as a non-root user with UID 1001 as a security measure, so the files and directories created by this pod also have an owner UID of 1001. However, since neither .root_user nor .root_password are group or world readable, and the CSI snapshot operation implemented via tar for NFS presumably runs as root which maps to nobody on the NFS share, the snapshot operation is unable to read these two files when running the tar command and fails.

There are at least two ways to work around this issue, but both involve reducing the overall security of our infrastructure and workloads and therefore unacceptable in a production environment:

Specifying no_root_squash for NFS which opens up the NFS share host to privilege escalation attacks
Configuring MinIO to run as root via specifying the appropriate Helm chart values which opens up the cluster to potential container escape and privilege escalation attacks

Conclusion

Don’t use NFS as a storage backend for your production-grade on-premises Kubernetes cluster!

+ Kubernetes, Storage

Configuring an IPsec VPN connection with OpenIKED

2023-09-16

+

+

+

+

+

+
From the Wikipedia page on virtual private networks (VPN):

A virtual private network (VPN) is a mechanism for creating a secure connection between a computing device and a computer network, or between two networks, using an insecure communication medium such as the public Internet.

VPNs are commonly used by businesses for enabling employees to work remotely e.g. during business trips or bad weather as they enable employees to connect to and access corporate resources securely from a remote network over the Internet. Another common use case for VPNs is for privacy-oriented individuals to conceal their true location and identity since all network requests are tunneled through the VPN and appear as if originating from the VPN server itself instead of the user’s laptop / workstation.

Now consider the following description of the IPsec protocol provided by Cloudflare:

IPsec is a group of protocols for securing connections between devices. IPsec helps keep data sent over public networks secure. It is often used to set up VPNs, and it works by encrypting IP packets, along with authenticating the source where the packets come from.

IPsec is commonly employed by enterprise-grade VPN solutions such as FortiGate for its excellent security and richness of features, though open-source implementations such as OpenIKED and strongSwan also exist. It uses Internet Key Exchange (IKE) under the hood which is further divided into IKEv1 and IKEv2, the latter a more modern and secure version of the former.

In the lab to follow, we will construct two disjoint networks on VirtualBox and demonstrate how OpenIKED can be configured as both a server and client to establish an IPsec VPN connection between the two networks.

Lab: Constructing two disjoint networks on VirtualBox and establishing a VPN connection between them with OpenIKED

Prerequisites

Familiarity with Unix/Linux networking and administration is assumed. If not, refer to my previous article Configuring a simple router with OpenBSD for recommended training and resources. It is also recommended that you follow and complete the lab in the previous article individually before proceeding with this lab, as the previous lab covers the basics of installing and configuring OpenBSD as a gateway device.

System requirements

A laptop / workstation with hardware virtualization enabled, a minimum of 2 CPU cores and 8G of RAM is required, though 16G or more RAM is recommended.

Overview

Here’s the network we’ll be constructing:

As illustrated in the diagram above, there are two networks moon-net and sun-net that cannot directly communicate with each other since they are separated by an “external” network cosmos and no routing rule exists between the two networks by default. Here, sun-net represents a corporate network where winnie hosts a web server only intended to be accessed internally by employees and sun is the gateway device acting as a router and firewall, while moon-net represents Alice’s home network with a personal laptop / workstation alice hidden behind a home router moon.

The requirement, then, is of course that Alice be able to establish a VPN connection to the corporate network and access the web server from the comfort of her home in order to perform her job duties as a diligent employee even when she is unable to return to office due to extreme weather.

To accurately model this network, we’ll leverage VirtualBox’s rich networking capabilities to construct moon-net and sun-net as internal networks and cosmos as a NAT network – the rest will be handled transparently by VirtualBox. We’ll then install and configure four virtual machines moon, sun, alice and winnie accordingly, using OpenBSD for our gateway devices moon, sun and Ubuntu Server for our client alice and web server winnie.

Initial VirtualBox networking configuration

The NAT network cosmos with subnet/netmask 192.168.100.0/24 needs to be created in advance, though the two internal networks moon-net and sun-net do not need to be explicitly created as VirtualBox groups together network interfaces connected to an internal network of identical name and automatically connects them to the same virtual switch which places them in the same virtual network.

To create the NAT network cosmos, select “Tools > Network > NAT Networks”, then click “Create” and fill in the details as shown in the next two screenshots:

Feel free to disable the DHCP server since we won’t be using it – instead, we will assign static IP addresses to all interfaces and hosts.

Creating and initializing our virtual machines

Let’s create and initialize our virtual machines in the following order, though note that the order of creation and initialization shouldn’t matter:
1. sun
2. moon
3. winnie
4. alice
Creating and installing OpenBSD on sun

Fetch the latest stable version of OpenBSD from the official website, then use that ISO to create a virtual machine named sun with 1024MB memory, 1 vCPU and 16G of storage.

Before booting sun for the first time, configure the settings for this VM as follows:
- Make sure “Enable I/O APIC” is unchecked under “System > Motherboard” as OpenBSD does not support paravirtualization and leaving this option checked is known to cause kernel panics
- For “Adapter 1” under “Network”, attach it to “NAT Network” and select the previously created network cosmos, ensuring that “Promiscuous Mode” is set to “Allow VMs” as well
- Make sure “Network > Adapter 2” is enabled and attached to the “Internal Network” sun-net
Click “OK” afterwards to save the settings and boot the VM normally.

Set the following parameters during installation:
- Hostname: sun
- em0 IPv4 address: 192.168.100.102
- em1 IPv4 address: 10.0.2.1
- Default route: 192.168.100.1
- Nameserver: 8.8.8.8
- Run X Window system: no
- Allow root SSH login: yes
- Continue without verification: yes
Remember to remove the installation ISO from the virtual drive explicitly before rebooting.

Creating and installing OpenBSD on moon

This is much the same as creating and installing OpenBSD on sun, except connect Adapter 2 to the internal network moon-net and set the following parameters during installation:
- Hostname: moon
- em0 IPv4 address: 192.168.100.101
- em1 IPv4 address: 10.0.1.1
- Default route: 192.168.100.1
- Nameserver: 8.8.8.8
- Run X Window system: no
- Allow root SSH login: yes
- Continue without verification: yes
Remember to remove the installation ISO from the virtual drive explicitly before rebooting.

Creating and installing Ubuntu Server on winnie

Fetch the latest LTS release of Ubuntu Server from the official website, then use that ISO to create a virtual machine winnie with 1024MB memory, 1 vCPU and 16G of storage. Make sure to skip the unattended installation which is known to fail and cause issues.

Before booting winnie for the first time, configure “Adapter 1” under “Network” such that it is connected to the internal network sun-net.

Click “OK” to save the modified settings.

Now boot the VM normally and set the following parameters during installation:
- IPv4 Method: Disabled
- Hostname: winnie
- Username: student
- Make sure “Install OpenSSH server” is checked
After the reboot, log in to the console directly (note that remote SSH login is not usable yet) and fill in the following content for /etc/netplan/00-installer-config.yaml:
# This is the network config written by 'subiquity' network: version: 2 ethernets: enp0s3: dhcp4: no addresses: - 10.0.2.2/24 gateway4: 10.0.2.1 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
Now run the following command to configure the network:
sudo netplan apply
Creating and installing Ubuntu Server on alice

This is similar to winnie, except connect Adapter 1 to the internal network moon-net and set the following parameters during installation:
- IPv4 Method: Disabled
- Hostname: alice
- Username: student
- Make sure “Install OpenSSH server” is checked
After the reboot, log in to the console directly and fill in the following content for /etc/netplan/00-installer-config.yaml:
# This is the network config written by 'subiquity' network: version: 2 ethernets: enp0s3: dhcp4: no addresses: - 10.0.1.2/24 gateway4: 10.0.1.1 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
Now run the following command to configure the network:
sudo netplan apply
Configuring host-level NAT for logging in to sun and moon via SSH

Since our gateway devices moon and sun are hidden behind a VirtualBox NAT router, we need to set the following NAT port forwarding rules at the host level to enable logging in to them remotely via SSH:
- Add a rule moon-ssh which forwards TCP port 1322 on the host to TCP port 22 on 192.168.100.101
- Add a rule sun-ssh which forwards TCP port 1222 on the host to TCP port 22 on 192.168.100.102
You may use any other ports on the host as desired, as long as they are both above 1024, distinct and not already bound by other services on your laptop / workstation.

With these port forwarding rules set, you should be able to start all VMs in headless mode and log in to them via SSH (or nested SSH with the appropriate gateway device as the intermediate host, for logging in to alice and winnie hidden behind an internal network).

Enable Internet access for both internal networks

Log in to sun via SSH (or directly via the console, or otherwise) and run the following commands to allow our “corporate firewall” to start forwarding packets:
sysctl net.inet.ip.forwarding=1 echo 'net.inet.ip.forwarding=1' >> /etc/sysctl.conf
Now append the following line to /etc/pf.conf:
pass out on em0 inet from 10.0.2.0/24 to any nat-to 192.168.100.102
Instruct PF to re-read the configuration file with the following command:
pfctl -f /etc/pf.conf
Now log in to moon via SSH and run the same commands:
sysctl net.inet.ip.forwarding=1 echo 'net.inet.ip.forwarding=1' >> /etc/sysctl.conf
Append the following line to /etc/pf.conf:
pass out on em0 inet from 10.0.1.0/24 to any nat-to 192.168.100.101
Instruct PF to re-read its configuration file:
pfctl -f /etc/pf.conf
Now log in to winnie with SSH via sun and confirm that it is able to reach the Internet:
ping -c4 google.com
Sample output:
PING google.com (172.217.24.78) 56(84) bytes of data. 64 bytes from hkg07s46-in-f14.1e100.net (172.217.24.78): icmp_seq=1 ttl=55 time=5.64 ms 64 bytes from hkg07s33-in-f14.1e100.net (172.217.24.78): icmp_seq=2 ttl=55 time=4.64 ms 64 bytes from sin10s06-in-f14.1e100.net (172.217.24.78): icmp_seq=3 ttl=55 time=4.74 ms 64 bytes from hkg07s46-in-f14.1e100.net (172.217.24.78): icmp_seq=4 ttl=55 time=4.86 ms --- google.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3004ms rtt min/avg/max/mdev = 4.637/4.968/5.637/0.394 ms
Repeat the same test with alice, by first logging in with SSH via moon.

With Internet access, we’re all set to install Apache on our web server winnie.

Install Apache on winnie

This step should be trivial:
sudo apt update && sudo apt install -y apache2
Confirm Apache is up and running:
curl -s localhost | grep 'It works!'
Sample output:
```
          It works!
```
Except now, note that alice is not able to reach winnie yet (run the command below on alice).
ping -c4 10.0.2.2
Sample output:
PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data. --- 10.0.2.2 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3055ms
Configuring an IPsec VPN connection between the two networks

For Alice to reach Winnie, we’ll need to establish a VPN connection between the two networks so they can see and communicate with each other directly.

An IPsec VPN connection consists of two components:
- The responder (server) listens for VPN connection requests in a passive manner
- The initiator (client) reaches out to a specified responder with a VPN connection request in an active manner
We’ll use OpenIKED for establishing our IPsec VPN connection which can be configured as both a responder and an initiator.

Configuring sun as a responder

Perform the steps below on sun to configure it as an IKEv2 responder. Note that OpenIKED only supports IKEv2 which is more modern and secure than IKEv1 anyway.

First open up some firewall rules allowing inbound connections from moon to UDP ports 500 (isakmp) and 4500 (ipsec-nat-t) as well as the ESP protocol. Append the following lines to /etc/pf.conf:
pass in log on em0 proto udp from 192.168.100.101 to 192.168.100.102 port {isakmp, ipsec-nat-t} tag IKED pass in log on em0 proto esp from 192.168.100.101 to 192.168.100.102 tag IKED
The OpenBSD FAQ on VPNs provides an explanation for these rules:

To enable the initiator to reach the responder, the isakmp UDP port should be open on the responder. If one of the peers is behind NAT, the ipsec-nat-t UDP port should also be open on the responder. If both peers have public IPs, then the ESP protocol should be allowed.

Allow PF to re-read its configuration file:
pfctl -f /etc/pf.conf
Next, fill in /etc/iked.conf as follows:
ikev2 'responder_psk' passive esp \ from 10.0.2.0/24 to 10.0.1.0/24 \ local 192.168.100.102 peer 192.168.100.101 \ psk 'P@ssw0rd'
Here, we instruct OpenIKED to listen for VPN connection requests in passive mode and construct a flow from sun-net (10.0.2.0/24) to moon-net (10.0.1.0/24) upon a successful VPN connection. The local “external” IP address is 192.168.100.102 as per our network diagram and the remote “external” IP address is that of moon, i.e. 192.168.100.101. For authentication, we use a pre-shared key (PSK) of P@ssw0rd which isn’t the most secure key or password out there but suffices for the purposes of this demo. Furthermore, it should be noted that a more secure method of authentication using RSA key pairs exists but it out of the scope of this lab, and PSKs are not that uncommon in real-world enterprise scenarios despite its weaker security.

Now ensure that /etc/iked.conf has its file permissions set to 0600, otherwise OpenIKED will simply refuse to start:
chmod 0600 /etc/iked.conf
Now start the iked service (OpenIKED):
rcctl -f start iked
Sample output:
iked(ok)
And enable iked to auto-start on system boot:
rcctl enable iked
Configuring moon as an initiator

The steps here are similar – fill in /etc/iked.conf with the appropriate configuration, set its file permissions to 0600, start the iked service and enable it to auto-start on system boot.

/etc/iked.conf:
ikev2 'initiator_psk' active esp \ from 10.0.1.0/24 to 10.0.2.0/24 \ peer 192.168.100.102 \ psk 'P@ssw0rd'
Notice that the initiator starts in active mode, from and to are swapped (since we’re on the other end of the VPN connection), local is not specified, the peer is swapped and the PSK is identical between the responder and initiator which is how the latter identifies itself to the former.

Now run the remaining commands on moon:
chmod 0600 /etc/iked.conf rcctl -f start iked rcctl enable iked
Verifying our IPsec VPN connection

Now run the following command on either sun or moon to view the IPsec flows:
ipsecctl -sa
Sample output:
FLOWS: flow esp in from 10.0.1.0/24 to 10.0.2.0/24 peer 192.168.100.101 srcid FQDN/sun.my.domain dstid FQDN/moon.my.domain type require flow esp out from 10.0.2.0/24 to 10.0.1.0/24 peer 192.168.100.101 srcid FQDN/sun.my.domain dstid FQDN/moon.my.domain type require SAD: esp tunnel from 192.168.100.101 to 192.168.100.102 spi 0x30d83f18 enc aes-128-gcm esp tunnel from 192.168.100.102 to 192.168.100.101 spi 0x7d37a775 enc aes-128-gcm
The first 3 lines of output demonstrate that bi-directional flows have been successfully established between the two subnets and hosts in both subnets should now be able to communicate with each other freely via the newly established VPN tunnel.

Let’s confirm this by logging in to both alice and winnie, and run some commands.

On winnie, ping the IP of alice to verify that it is reachable:
ping -c4 10.0.1.2
Sample output:
PING 10.0.1.2 (10.0.1.2) 56(84) bytes of data. 64 bytes from 10.0.1.2: icmp_seq=1 ttl=62 time=1.58 ms 64 bytes from 10.0.1.2: icmp_seq=2 ttl=62 time=1.46 ms 64 bytes from 10.0.1.2: icmp_seq=3 ttl=62 time=2.54 ms 64 bytes from 10.0.1.2: icmp_seq=4 ttl=62 time=1.75 ms --- 10.0.1.2 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms rtt min/avg/max/mdev = 1.455/1.831/2.536/0.420 ms
Now, on alice, ping the IP address of winnie to verify that it is reachable:
ping -c4 10.0.2.2
Sample output:
PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data. 64 bytes from 10.0.2.2: icmp_seq=1 ttl=62 time=0.575 ms 64 bytes from 10.0.2.2: icmp_seq=2 ttl=62 time=0.776 ms 64 bytes from 10.0.2.2: icmp_seq=3 ttl=62 time=1.83 ms 64 bytes from 10.0.2.2: icmp_seq=4 ttl=62 time=2.50 ms --- 10.0.2.2 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3041ms rtt min/avg/max/mdev = 0.575/1.419/2.496/0.783 ms
Finally and most importantly, confirm that Alice is able to visit the internal webpage hosted by Winnie:
curl -s 10.0.2.2 | grep 'It works!'
Sample output:
```
          It works!
```
Concluding remarks and going further

We’ve seen how OpenIKED can be configured as both an IPsec VPN server (responder) and client (initiator) for establishing VPN connections in a semi real-world scenario, but note that there are two major caveats:
- This lab assumes that all hosts and interfaces are assigned static IP addresses and thus known in advance, though in a real-world scenario, the public IP of a home router is likely dynamically allocated by an Internet Service Provider (ISP) via DHCP, and similarly, the private IP of alice will likely be dynamically allocated via DHCP as well
- This lab configures both the IKEv2 responder and initiator on the gateway devices moon and sun, though in a real-world scenario, Alice will likely configure the VPN client directly on her laptop alice in a road warrior configuration and not directly on her home router moon
While this lab was hopefully interesting and educational, it only covers the most basic use case for OpenIKED and there is much left to uncover. For a deeper dive in OpenIKED, consider going through the remaining sections of the OpenBSD FAQ on VPNs (this lab only touches the first section) and adapt this lab for the road warrior configuration which is closer to how IPsec VPNs are typically configured and used in a real-world scenario.

I hope you enjoyed this article and see you in a future article 😉
+

+

+

+

+

+

+ Networking

+

+

+

+

+

+

+

+

+

+

+

+
Configuring a simple router with OpenBSD

2023-09-12

+

+

+

+

+

+
OpenBSD is a security-first Unix-like operating system belonging to the BSD family of operating systems. It is best known for the OpenSSH project founded in 1999 under the OpenBSD umbrella which has garnered widespread adoption beyond OpenBSD, including the infamous Linux operating system and even Microsoft Windows (since Windows 10).

While industrial adoption of OpenBSD is nowhere near as prevalent as Linux, it is nonetheless well-suited for use in self-managed firewalls and routers for individual technical enthusiasts and small to medium businesses alike, where product warranty, commercial support and compliance is not a major issue. Furthermore, unlike the industrial behemoth that Linux has become, OpenBSD has remained true to the Unix philosophy over the years which emphasizes the principle:

Do one thing, and do it well.

This is reflected in the overall lean, coherent design of the OpenBSD operating system featuring short, succint commands and configuration files that concisely perform their intended function with minimal syntactical and cognitive overhead, which makes it an arguably better choice than Linux for learning the basics of computer networking and security.

In the lab to follow, we will configure OpenBSD as a router for forwarding requests to the Internet and expose a traditional LAMP server to external clients via port forwarding with NAT. Don’t worry, no additional hardware equipment and setup is required – all you need for this lab is a laptop or workstation capable of running VirtualBox 🙂

Lab: Modeling a simple network with a single LAMP server and router

Here’s a simple illustration of the network we’ll be building, with a web server placed behind a router:

Prerequisites

Familiarity with the Unix/Linux command line is assumed. If not, consider enrolling in LFS101x: Introduction to Linux which is offered by The Linux Foundation on edX at no cost.

A basic theoretical understanding of computer networking and communications is assumed. If not, consider studying an undergraduate-level textbook on the subject matter such as Computer Networking: A Top-Down Approach.

Last but not least, you should also be familiar with the concept of virtual machines and ideally experience in using a consumer-grade hypervisor such as VirtualBox.

System requirements

A laptop or workstation capable of running VirtualBox with hardware virtualization enabled, at least 2 CPU cores and 8G of RAM, though 16G+ of RAM is preferable.

It is assumed VirtualBox is already installed – if not, go ahead to download and install it.

Setting up our virtual router with OpenBSD

The latest stable version of OpenBSD at the time of writing is 7.3. Visit the downloads page and select install73.iso for the amd64 architecture.

Now create a new virtual machine with install73.iso for the installation disk and name it openbsd or any other name as desired. Feel free to leave the other settings at their defaults – 1024MB RAM, 1 vCPU and 16G of storage should suffice.

Before booting our virtual machine for the first time, go to the Settings page for this virtual machine and change the settings below:
- Ensure the “Enable I/O APIC” option is unchecked under “System > Motherboard” as leaving this option checked is known to cause kernel panics on OpenBSD
- Under “Network > Adapter 2”, ensure the second network adapter is enabled and attached to “Internal Network”, leaving the network name as the default “intnet”
Remember to click “OK” at the bottom of the window to save your changes once the configuration is complete.

Now boot the virtual machine and follow the prompts to install OpenBSD, accepting the default options unless otherwise specified. Some options to take note of:
- Network interface em0 corresponds to “Adapter 1” and em1 to “Adapter 2” under the “Network” section in the virtual machine settings. As per our simplified network diagram above, since em0 is connected to the “external” network and em1 to the internal network “intnet”, set em0‘s IP address to 10.0.2.15 and em1‘s IP address to 192.168.100.1, both with a default netmask of 255.255.255.0. Leave their IPv6 addresses blank and accept the default symbolic host name for em1
- Set the default route to 10.0.2.2 – see User Networking (SLIRP) for a detailed explanation
- Set the DNS nameserver to 8.8.8.8 – this IP address corresponds to Google DNS
- Answer “no” when prompted to run the X Window system since we won’t be needing a graphical interface
- You may wish to enable root SSH login for convenience in this demo, though note that this shouldn’t be enabled in production
- When reminded that “directory does not contain SHA256.sig”, answer “yes” to continue without verification. While not the most secure option, it shouldn’t be a major issue for the purposes of this demo
Once prompted to reboot in order to finish the installation, remember to “Remove Disk from Virtual Drive” under the “Storage” section within the VM settings as shown in the screenshot below bebore proceeding with the reboot.

After the reboot, optionally configure port forwarding to forward SSH (TCP port 22) from your OpenBSD router to your local laptop / workstation for a more convenient command-line experience. You may also wish to boot the router in headless mode if connecting via SSH so VirtualBox doesn’t open a new window for the VM every time it boots.

Installing Ubuntu for our web server

Create a VM for installing Ubuntu as usual, though keep note of the following:
- Install the server edition since we won’t be needing a desktop environment
- Make sure to skip the unattended installation which is known to fail and cause issues
- Allocating 1024MB of RAM and 1 vCPU should suffice, or leave them at their defaults
- 16G of storage should suffice, or leave it at the default
- Under “Settings > Network”, disable “Adapter 1” as shown in the screenshot below – we’ll configure the network adapter separately and enable Internet access after the installation
- During the OS installation, make sure “Install OpenSSH server” is selected so we can later start our web server in headless mode and configure it remotely via SSH
Once the installation is complete, shut down the VM and re-enable “Adapter 1” under “Settings > Network”, attaching it to “Internal Network” and leaving the name as the default intnet.

Now boot the VM normally (note that SSH is not usable yet), log in to the console and fill in /etc/netplan/00-installer-config.yaml with the following content:
network: version: 2 ethernets: enp0s3: dhcp4: no addresses: - 192.168.100.2/24 gateway4: 192.168.100.1 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
This sets the IP address of our web server to 192.168.100.2 with our router 192.168.100.1 as the gateway and Google DNS for the nameservers.

Save the file and exit, then apply the changes with the following command:
sudo netplan apply
You may see a few warnings – as long as there are no errors, you should be fine 🙂

Testing initial network connectivity for our web server and router

Now ensure both our Ubuntu web server and OpenBSD router are up and running, then proceed to log in to the OpenBSD router as the root user.

Confirm that our OpenBSD router is able to reach the Internet:
ping -c4 google.com
Sample output:
PING google.com (142.250.204.46): 56 data bytes 64 bytes from 142.250.204.46: icmp_seq=0 ttl=254 time=4.935 ms 64 bytes from 142.250.204.46: icmp_seq=1 ttl=254 time=6.977 ms 64 bytes from 142.250.204.46: icmp_seq=2 ttl=254 time=7.116 ms 64 bytes from 142.250.204.46: icmp_seq=3 ttl=254 time=7.138 ms --- google.com ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 4.935/6.541/7.138/0.930 ms
Confirm also that our router is able to reach the web server:
ping -c4 192.168.100.2
Sample output:
PING 192.168.100.2 (192.168.100.2): 56 data bytes 64 bytes from 192.168.100.2: icmp_seq=0 ttl=64 time=0.489 ms 64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=1.020 ms 64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=0.915 ms 64 bytes from 192.168.100.2: icmp_seq=3 ttl=64 time=0.956 ms --- 192.168.100.2 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.489/0.845/1.020/0.209 ms
Now assume the user you created on the web server is named student. SSH into the web server via our router – replace the placeholder student as appropriate:
ssh student@192.168.100.2
Assuming the SSH login was successful, you should now see the shell prompt change to that of the web server.

Inside our web server, ping the internal IP address of our router 192.168.100.1 to confirm once more that it is reachable from our web server (though technically logging in from our router via SSH already proves that):
ping -c4 192.168.100.1
Sample output:
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data. 64 bytes from 192.168.100.1: icmp_seq=1 ttl=255 time=0.124 ms 64 bytes from 192.168.100.1: icmp_seq=2 ttl=255 time=0.268 ms 64 bytes from 192.168.100.1: icmp_seq=3 ttl=255 time=0.282 ms 64 bytes from 192.168.100.1: icmp_seq=4 ttl=255 time=0.609 ms --- 192.168.100.1 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3057ms rtt min/avg/max/mdev = 0.124/0.320/0.609/0.177 ms
However, note that our web server currently has no Internet connectivity:
ping -c4 google.com
Sample output:
ping: google.com: Temporary failure in name resolution
Don’t worry – we’re going to change that in a moment 😉

Enabling Internet connectivity for our web server

Our web server can’t reach the Internet since our router is not configured to forward traffic from our internal network to the Internet by default, so we just need to configure our router accordingly.

OpenBSD comes with a packet filter PF built into the kernel for the purposes of network routing, firewalls, network address translation (NAT), etc., which can be queried and configured via the pfctl command-line utility. For those familiar with the Linux networking stack, it’s essentially OpenBSD’s equivalent to iptables / nftables.

Take some time to go through the official PF introduction before we proceed.

Now set the kernel parameter net.inet.ip.forwarding to 1 and persist it on system boot so our router is able to forward any packets at all:
sysctl net.inet.ip.forwarding=1 echo 'net.inet.ip.forwarding=1' >> /etc/sysctl.conf
Next, log in to our router and append the following line to /etc/pf.conf, the main configuration file for PF:
pass out on em0 inet from 192.168.100.0/24 to any nat-to 10.0.2.15
Hint: the vi editor is available by default on OpenBSD

This line instructs PF to forward all requests from our internal network 192.168.100.0/24 through our router’s external interface em0, replacing the source IP address with our router’s external IP address 10.0.2.15 via NAT, regardless of the destination.

Now instruct PF to re-read our configuration file:
pfctl -f /etc/pf.conf
Back to our web server, notice that it is now able to connect to the Internet:
ping -c4 google.com
Sample output:
PING google.com (142.251.220.110) 56(84) bytes of data. 64 bytes from hkg07s52-in-f14.1e100.net (142.251.220.110): icmp_seq=1 ttl=55 time=5.51 ms 64 bytes from hkg07s52-in-f14.1e100.net (142.251.220.110): icmp_seq=2 ttl=55 time=4.90 ms 64 bytes from hkg07s52-in-f14.1e100.net (142.251.220.110): icmp_seq=3 ttl=55 time=5.06 ms 64 bytes from hkg07s52-in-f14.1e100.net (142.251.220.110): icmp_seq=4 ttl=55 time=4.42 ms --- google.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3018ms rtt min/avg/max/mdev = 4.416/4.971/5.513/0.392 ms
Congratulations, our web server is now able to reach the Internet!

Installing Apache on our web server

Now that our web server is able to reach the Internet, let’s install Apache and get our website up and running which should be trivial – simply install the apache2 package from the system repositories:
sudo apt update && sudo apt install -y apache2
Once installed, Apache should be enabled out of the box – let’s confirm this by visiting the website locally on our web server:
curl -s localhost | grep 'It works!'
Sample output:
```
          It works!
```
Our website should also be reachable from our router via the internal IP 192.168.100.2 of our web server, but first install curl explicitly on our router which is not installed by default on OpenBSD:
pkg_add curl
Now curl the internal IP of our web server and confirm that it works:
curl -s 192.168.100.2 | grep 'It works!'
Sample output:
```
          It works!
```
But now we want to expose our website to the Internet, which is currently not externally reachable since our web server is hidden behind an internal network. Fortunately, the solution is simple – we just need to forward the appropriate port on our router via NAT.

Exposing our website to the Internet

To ensure our website is reachable from external clients, i.e. those outside our private network 192.168.100.0/24, we need to expose an appropriate port on our router and forward requests from that port to the appropriate port on our web server. Since our Apache web server is a typical HTTP web server, we’ll need to forward requests to port 80 on our web server and let’s decide to expose this as port 8080 on our router.

So logically, we expect external clients to visit our website via the URL http://10.0.2.15:8080/ through a web browser (or curl, or otherwise), and then our router will translate this to http://192.168.100.2/ behind the scenes (the default TCP port 80 used for HTTP is implied), though our visitors should neither know nor care about this.

To enable this behavior, we need to append the following line to /etc/pf.conf on our router:
pass in on em0 proto tcp from any to 10.0.2.15 port 8080 rdr-to 192.168.100.2 port 80
Now instruct PF to re-read our configuration file once again:
pfctl -f /etc/pf.conf
With this port forwarding rule in place, let’s simulate a request from an external client. In our case, this will simply be the VirtualBox host, i.e. the laptop / workstation you are following this lab on.

Unfortunately, our laptop / workstation can’t actually see the IP address 10.0.2.15 directly since our virtual router is actually hidden behind the VirtualBox default NAT router as illustrated below:

So we need to configure an additional NAT port forwarding rule on the host level. For simplicity, let’s forward port 8080 on our host to port 8080 on our router.

To do this, go to our router VM “Settings > Network > Adapter 1 > Advanced” and select “Port Forwarding” as shown in the screenshot below:

Next, add a rule to forward TCP port 8080 on the VirtualBox host to TCP port 8080 on our virtual router as shown below and click “OK” to confirm the changes:

Don’t forget to click “OK” one more time to save the modified settings.

Now on your laptop / workstation, visit http://localhost:8080/ and voila – you should see Ubuntu’s version of the default Apache web page:

Concluding remarks and going further

We saw in this lab how to configure OpenBSD as a router using the built-in PF packet filter to forward requests from an internal network to the Internet, and how to configure NAT port forwarding with PF to expose internal services to external clients.

While this lab was (hopefully) fun and interesting, it barely scratches the surface of what OpenBSD is capable of so stay tuned for more content 😉
+

+

+

+

+

+

+ Networking

+

+

+

+

+

+

+

+

+

+

+

+

Kafka architecture

Lab: Deploy Kafka to Docker and Kubernetes

Prerequisites

Setting up your environment

Install Docker and Docker Compose V2

Install kind, kubectl and Helm for Kubernetes

Install cloud-provider-kind for load balancing

Deploy Kafka to Docker with Docker Compose V2

Deploy Kafka to Kubernetes with Strimzi

Prepare a private root CA

Start your Kubernetes cluster

Install the Kubernetes Ingress NGINX controller

Get the IP of our ingress controller and prepare a TLS serving certificate

Deploy the Strimzi operator with Helm

Deploy a Strimzi Kafka cluster with 1 controller and 1 broker

Concluding remarks and going further

Lab: Enforce Kubernetes compliance controls and mitigate vulnerabilities with ValidatingAdmissionPolicy

Prerequisites

Setting up your environment

Install Docker

Install kind and spin up a cluster

Install and configure kubectl

Enforce CIS Kubernetes Benchmarks with ValidatingAdmissionPolicy

Mitigate CVE-2024-10220 with ValidatingAdmissionPolicy

Concluding remarks and going further

Lab: Back up and restore WordPress on Kubernetes with Kanister

Prerequisites

Setting up your environment

Configure your AWS credentials

Install OpenTofu

Install Docker

Install kind and spin up a cluster

Install and configure kubectl

Install and configure Helm

Install Go 1.22

Install the Kanister tools

Installing WordPress

Installing the Kanister operator

Creating our S3 bucket for storing our Kanister backups

Backing up our WordPress database to S3

“Accidentally” deleting our Hello World blog post

Restoring our WordPress database from S3

Concluding remarks and going further

System requirements

Generating certificates for SSL/TLS

Deploying our Keycloak instance with Podman

Creating a realm for our organization

Enforcing mandatory MFA for all users in our realm

Creating an account for John Doe

Creating a client for the Keycloak SPA demo application

Installing the Authenticator MFA app on Flathub

Logging in as John Doe with SSO

Cleaning up

Concluding remarks and going further

Lab: Exposing your MicroK8s homelab to the Internet

Prerequisites

Setting up your environment

Generating an SSH key pair for our EC2 instance

Setting up your AWS account

Setting up AWS CLI v2

Installing OpenTofu

Deploying the automation stack

Preparing our EC2 instance for reverse tunneling

Copying artifacts to our MicroK8s edge node

Installing MicroK8s on our edge node

Creating our SSH reverse tunnel with AutoSSH

Setting up our website with SSL/TLS using cert-manager and Let’s Encrypt

Tearing down

Concluding remarks and going further

Lab: Provisioning a single-node OKD cluster using CRC

Prerequisites

Setting up your environment

Spinning up an OpenShift Local cluster

Exploring our cluster with the kubectl oc command line

Exploring our cluster with the OpenShift web console

Setting up Argo CD with the Operator

Cleaning up

Concluding remarks and going further

Lab: Deploying Rook Ceph to a 3-node kubeadm cluster

Prerequisites

Install `cloud-provider-kind` for load balancing

Exploring our cluster with the `kubectl` `oc` command line

`ansible/ansible.cfg`

`ansible/hosts.yaml`

Installing MinIO and creating a `VolumeSnapshot`

Using `nfs-subdir-external-provisioner` instead of `csi-driver-nfs`

Setting up `master0`

Setting up `worker0`

Attempt to create a snapshot from the `minio` PVC

Creating and installing OpenBSD on `sun`

Creating and installing OpenBSD on `moon`

Creating and installing Ubuntu Server on `winnie`

Creating and installing Ubuntu Server on `alice`

Configuring host-level NAT for logging in to `sun` and `moon` via SSH

Install Apache on `winnie`

Configuring `sun` as a responder

Configuring `moon` as an initiator