Enforcing recommended practices on Kubernetes with ValidatingAdmissionPolicy

Date: 2024-12-28

Recall that each Kubernetes request goes through 3 stages as it is evaluated by the API server:

1. Authentication: who you are
2. Authorization: what actions are allowed
3. Admission control: additional validation and mutation checks before the finalized request is accepted (rejected)

See also: Controlling Access to the Kubernetes API | Kubernetes

While many built-in admission controllers are enabled by default on Kubernetes, it was already possible since the early days of Kubernetes to extend admission control by writing your own admission webhooks. However, a drawback to this approach is that it requires some degree of programming knowledge which not all Kubernetes administrators may readily possess.

To simplify the process of implementing custom admission control logic, ValidatingAdmissionPolicy was introduced as an alpha feature in Kubernetes 1.26 and promoted to GA in 1.30. It allows administrators to easily and quickly define request validation logic with Common Expression Language (CEL) instead of writing code. This lowers the barrier for administrators to enforce compliance controls and recommended practices as well as mitigate platform vulnerabilities in Kubernetes environments.

This lab exercise demonstrates the use of ValidatingAdmissionPolicies to perform the following tasks:

1. Disallow pods in the default namespace as per CIS Kubernetes 5.7.4
2. Mitigate CVE-2024-10220 by rejecting pods with gitRepo volumes

Lab: Enforce Kubernetes compliance controls and mitigate vulnerabilities with ValidatingAdmissionPolicy

This lab has been tested with Kubernetes v1.32 (Penelope).

Prerequisites

Familiarity with Kubernetes is assumed. See LFS158x: Introduction to Kubernetes for a gentle introduction to Kubernetes.

Setting up your environment

A Linux environment with at least 2 vCPUs, 8GiB memory and sufficient available disk space capable of running Docker. This can be your own desktop/laptop if you’re a Linux user (like I am ;-), or a spare board (e.g. Raspberry Pi), physical server, virtual machine or cloud instance.

The reference environment is Ubuntu 24.04 LTS (Noble Numbat) so if you’re on a different Linux distribution, adapt apt-related commands with dnf / pacman / something else accordingly when installing system packages. Otherwise, the remaining instructions should be broadly applicable to most Linux distributions.

Install Docker

We’ll use Docker to spin up a kind Kubernetes cluster. It’s convenient, fast, simple and sufficient for this lab exercise.

Install the Docker engine and add the current user to the docker group:

sudo apt update && sudo apt install -y docker.io
sudo usermod -aG docker "${USER}"

Log out and in for the changes to take effect.

Check that we have the correct version of Docker installed:

docker version

Sample output:

Client:
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.22.2
 Git commit:        26.1.3-0ubuntu1~24.04.1
 Built:             Mon Oct 14 14:29:26 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.22.2
  Git commit:       26.1.3-0ubuntu1~24.04.1
  Built:            Mon Oct 14 14:29:26 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.12
  GitCommit:        
 runc:
  Version:          1.1.12-0ubuntu3.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:

Install kind and spin up a cluster

Just follow the instructions in their Quickstart:

# For AMD64 / x86_64
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.26.0/kind-linux-amd64
# For ARM64
[ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.26.0/kind-linux-arm64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

Check the correct kind version is installed:

kind version

Sample output:

kind v0.26.0 go1.23.4 linux/amd64

Now our Kubernetes cluster is but a single command away:

kind create cluster

Install and configure kubectl

Again, the official instructions will suffice:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/.

Check that kubectl is correctly installed:

kubectl version

Sample output:

Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.32.0

For command-line completion, add the following line to your ~/.bashrc:

source <(kubectl completion bash)

Now save the file and run:

source ~/.bashrc

Enforce CIS Kubernetes Benchmarks with ValidatingAdmissionPolicy

The CIS Kubernetes Benchmarks includes the following rule which typically requires a manual check:

5.7.4. The default namespace should not be used

Fortunately, we can enforce this rule automatically as a CEL expression with ValidatingAdmissionPolicy. The trivial CEL expression false rejects all requests unconditionally and we can scope its evaluation to Pods only.

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: deny-all-pods
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: "false"

Save the policy above as deny-all-pods.yaml and apply it with kubectl apply.

kubectl apply -f deny-all-pods.yaml

Sample output:

validatingadmissionpolicy.admissionregistration.k8s.io/deny-all-pods created

View the created ValidatingAdmissionPolicy:

kubectl get validatingadmissionpolicies

Sample output:

NAME            VALIDATIONS   PARAMKIND   AGE
deny-all-pods   1             <unset>     7s

Next, apply the policy to the default namespace with the ValidatingAdmissionPolicyBinding below.

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: deny-all-pods-default
spec:
  policyName: deny-all-pods
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: default

Save it as deny-all-pods-default.yaml and apply it to the cluster.

kubectl apply -f deny-all-pods-default.yaml

Sample output:

validatingadmissionpolicybinding.admissionregistration.k8s.io/deny-all-pods-default created

View the created ValidatingAdmissionPolicyBinding:

kubectl get validatingadmissionpolicybindings

Sample output:

NAME                    POLICYNAME      PARAMREF   AGE
deny-all-pods-default   deny-all-pods   <unset>    2m8s

Now confirm that we are unable to create pods in the default namespace. Save the YAML below as busybox.yaml:

---
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: busybox
  name: busybox
spec:
  containers:
    - args:
        - sleep
        - infinity
      image: busybox
      name: busybox
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Try to create the pod with kubectl create and notice how the request is rejected.

kubectl create -f busybox.yaml

Sample output:

The pods "busybox" is invalid: : ValidatingAdmissionPolicy 'deny-all-pods' with binding 'deny-all-pods-default' denied request: failed expression: false

Let’s confirm that we are able to create pods in other namespaces. Create a busybox namespace:

kubectl create ns busybox

Now save the YAML below as busybox-busybox.yaml which defines the same Busybox pod but in the busybox namespace instead of the default namespace.

---
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: busybox
  name: busybox
  namespace: busybox
spec:
  containers:
    - args:
        - sleep
        - infinity
      image: busybox
      name: busybox
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Create the pod with kubectl create and confirm that the pod is created.

kubectl create -f busybox-busybox.yaml

Sample output:

pod/busybox created

Now confirm the pod is running.

kubectl -n busybox get pods

Sample output:

NAME      READY   STATUS    RESTARTS   AGE
busybox   1/1     Running   0          58s

Delete the pod - we’ll re-create it afterwards.

kubectl delete -f busybox-busybox.yaml

Mitigate CVE-2024-10220 with ValidatingAdmissionPolicy

CVE-2024-10220 is a high severity vulnerability with real-world exploits by leveraging gitRepo volumes deprecated since Kubernetes 1.11.

The Kubernetes kubelet component allows arbitrary command execution via specially crafted gitRepo volumes.

While this vulnerability was patched in Kubernetes 1.31 and backported to 1.29.7 and 1.30.3, Kubernetes upstream recommends replacing existing gitRepo volumes with emptyDir volumes initialized via init containers and there is no reason for legitimate workloads to use gitRepo volumes today. Therefore, it would make sense to define a ValidatingAdmissionPolicy and corresponding ValidatingAdmissionPolicyBinding to disallow the use of gitRepo volumes entirely across all namespaces.

The CEL expression for rejecting gitRepo volumes below is taken from the official documentation.

!has(object.spec.volumes) || !object.spec.volumes.exists(v, has(v.gitRepo))

The ValidatingAdmissionPolicy - name it forbid-gitrepo-volumes.yaml:

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: forbid-gitrepo-volumes
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: "!has(object.spec.volumes) || !object.spec.volumes.exists(v, has(v.gitRepo))"

The associated ValidatingAdmissionPolicyBinding - name it forbid-gitrepo-volumes-binding.yaml:

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: forbid-gitrepo-volumes-binding
spec:
  policyName: forbid-gitrepo-volumes
  validationActions: [Deny]

Create the policy and associated binding.

kubectl create -f forbid-gitrepo-volumes.yaml
kubectl create -f forbid-gitrepo-volumes-binding.yaml

Confirm we can create our Busybox pod in the busybox namespace since it does not contain a gitRepo volume:

kubectl create -f busybox-busybox.yaml

Sample output:

pod/busybox created

Now prepare a modified Busybox pod with a gitRepo volume - name the file busybox-gitrepo-busybox.yaml.

---
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: busybox
  name: busybox-gitrepo
  namespace: busybox
spec:
  containers:
    - args:
        - sleep
        - infinity
      image: busybox
      name: busybox
      resources: {}
      volumeMounts:
        - name: my-gitrepo-volume
          mountPath: /srv/source
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  volumes:
    - name: my-gitrepo-volume
      gitRepo:
        repository: git@github.com:kubernetes/kubernetes
        revision: 7bfdda4696f78fc789fe91420f7c8609c71002d0
status: {}

Try to create the pod with kubectl create and observe the failure.

kubectl create -f busybox-gitrepo-busybox.yaml

Sample output:

Warning: spec.volumes[0].gitRepo: deprecated in v1.11
The pods "busybox-gitrepo" is invalid: : ValidatingAdmissionPolicy 'forbid-gitrepo-volumes' with binding 'forbid-gitrepo-volumes-binding' denied request: failed expression: !has(object.spec.volumes) || !object.spec.volumes.exists(v, has(v.gitRepo))

Concluding remarks and going further

While ValidatingAdmissionPolicy allows Kubernetes administrators to define resource validation logic with CEL expressions without writing code, MutatingAdmissionPolicy was recently introduced in Kubernetes 1.32 as an alpha feature allowing resource mutation logic to be similarly defined as a combination of CEL expressions and JSON patches without writing code. However, as an alpha feature, it is disabled by default and hidden behind a feature gate which must be explicitly enabled.

Kubernetes-native admission policies enable Kubernetes administrators to enforce policies with ease at the cluster, namespace and resource level. While they serve as a ready replacement for admission webhooks and perhaps even some simple use cases of policy engines such as OPA Gatekeeper, third-party policy engines such as OPA Gatekeeper, Kyverno and Kubewarden often provide additional features and guarantees beyond the scope of upstream Kubernetes useful for seasonsed Kubernetes administrators:

• Integrated CLI support
• Graphical user interface
• Advanced policy reporting
• Enterprise support plans
• … etc.

Subscribe:

Return to homepage