Using Pod Security Admission with Kubewarden

Starting from Kubernetes 1.25 release, the Pod Security Policies have been removed and have been replaced by Pod Security Admission, also known as "PSA".

Pod Security Admission has been created to simplify the process of securing Pods defined inside of Kubernetes clusters. Because of that it doesn't expose the same amount of configuration knobs provided by Pod Security Policies.

Pod Security Admission works at the Namespace level and provides three profiles that can be applied: privileged, baseline and restricted. The privileged profile has few limitations, while the restricted one is the one that tightens Pod permissions.

In addition to choosing the profile to be used, the Kubernetes operators can decide what kind of action is going to be performed by Pod Security Admission controller once a violation is detected. The possible actions are: enforce, audit and warn.

Pod Security Admission is not a full replacement for the old Pod Security Policies. Currently, with the Kubernetes 1.25 release, it suffers the following limitations:

No mutation capabilities
Higher level objects (like Deployment, Job) are evaluated only when the audit or warn modes are enabled

A solution like Kubewarden can be used to integrate a Pod Security Admission profile to workaround these limitations.

note

Kubewarden can be used to replace all the previous Pod Security Policies as shown here. However, the goal of this article is to demonstrate how Kubewarden can complement the new Pod Security Admission.

Example

Now let's take a look in the example creating a Namespace where the most restrictive Pod Security Admission policies are enforced:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.25
EOF

This PSA profile does not allow the creation of containers that run their application as root user. In other words, when defining a container, the runAsNonRoot attribute must be set to true and the runAsUser one cannot be set to 0.

Therefore, the following resource will not reach its desired state:

kubectl apply -n my-namespace -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: template-nginx
  template:
    metadata:
      labels:
        app: template-nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        securityContext:
          runAsNonRoot: true
          runAsUser: 0
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - "ALL"
          seccompProfile:
            type: "RuntimeDefault"
        ports:
        - containerPort: 80
EOF

If we check the Deployment, we can see the Pod Security Admission is preventing its Pod from being created:

kubectl get deploy -n my-namespace nginx-deployment -o json | jq ".status.conditions[] | select(.reason == \"FailedCreate\")"
{
  "lastTransitionTime": "2022-10-28T19:09:56Z",
  "lastUpdateTime": "2022-10-28T19:09:56Z",
  "message": "pods \"nginx-deployment-5f98b4db8c-2m96l\" is forbidden: violates PodSecurity \"restricted:v1.25\": runAsUser=0 (container \"nginx\" must not set runAsUser=0)",
  "reason": "FailedCreate",
  "status": "True",
  "type": "ReplicaFailure"
}

The user can fix this issue removing the runAsUser from the container definition:

kubectl apply -n my-namespace -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: template-nginx
  template:
    metadata:
      labels:
        app: template-nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        securityContext:
          runAsNonRoot: true
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - "ALL"
          seccompProfile:
            type: "RuntimeDefault"

        ports:
        - containerPort: 80
EOF

This time the Pod Security Admission will allow the Pod to be created, but the Pod creation will still fail.

kubectl get pods -n my-namespace
NAME                                READY   STATUS                       RESTARTS   AGE
nginx-deployment-57d8568bbb-h4bx7   0/1     CreateContainerConfigError   0          47s

That's happening because the container definition didn't specify which user should be used when starting a program inside of the container. That results with the the root user (0) being used, which is not allowed by the runAsNonRoot directive:

kubectl get pods -n my-namespace nginx-deployment-57d8568bbb-h4bx7 -o json | jq ".status.containerStatuses"
[
  {
    "image": "nginx:1.14.2",
    "imageID": "",
    "lastState": {},
    "name": "nginx",
    "ready": false,
    "restartCount": 0,
    "started": false,
    "state": {
      "waiting": {
        "message": "container has runAsNonRoot and image will run as root (pod: \"nginx-deployment-57d8568bbb-8mvkc_my-namespace(add7bcc5-3d23-43d0-94e9-6e78f887a53f)\", container: nginx)",
        "reason": "CreateContainerConfigError"
      }
    }
  }
]

This is where Kubewarden can help. In this case the [user-group-policy] (https://artifacthub.io/packages/kubewarden/user-group-psp/user-group-psp) policy can be used to mutate the Deployment definition, configuring a default user to be used to all the containers that omit this information. Or even enforcing a user ID range. Let's apply the policy: The policy even allows to set a user ID range:

note

This example assumes the Kubewarden stack is already installed inside of your Kubernetes cluster. See the QuickStart article for more details.

kubectl apply -f - <<EOF
apiVersion: policies.kubewarden.io/v1
kind: ClusterAdmissionPolicy
metadata:
  name: user-group-psp
spec:
  policyServer: default
  module: registry://ghcr.io/kubewarden/policies/user-group-psp:latest
  rules:
  - apiGroups: ["", "apps"]
    apiVersions: ["v1"]
    resources: ["pods", "deployments"]
    operations:
    - CREATE
    - UPDATE
  mutating: true
  settings:
    run_as_user:
      rule: "MustRunAs"
      overwrite: false
      ranges:
        - min: 1000
          max: 2000
        - min: 4000
          max: 5000
    run_as_group:
      rule: "RunAsAny"
    supplemental_groups:
      rule: "RunAsAny"
EOF

Before moving forward, we have to wait for the policy to be active. This can be checked with the following command:

kubectl get clusteradmissionpolicy.policies.kubewarden.io/user-group-psp

Once the policy is active, we can re-create the Deployment:

kubectl delete deployment -n my-namespace nginx-deployment && \
kubectl apply -n my-namespace -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: template-nginx
  template:
    metadata:
      labels:
        app: template-nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        securityContext:
          runAsNonRoot: true
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - "ALL"
          seccompProfile:
            type: "RuntimeDefault"

        ports:
        - containerPort: 80
EOF

This time the Deployment definition is mutated by Kubewarden's policy, allowing the Pod to be started. The container defined inside of the Pod will have a default runAsUser value:

kubectl get pods -n my-namespace nginx-deployment-57d8568bbb-nv8fj -o json | jq ".spec.containers[].securityContext"
{
  "allowPrivilegeEscalation": false,
  "capabilities": {
    "drop": [
      "ALL"
    ]
  },
  "runAsNonRoot": true,
  "runAsUser": 1000,
  "seccompProfile": {
    "type": "RuntimeDefault"
  }
}

The Kubewarden integration can do even more in this scenario: it can validate the value of the runAsUser provided by the users.

For example, the following resource will be reject by the Kubewarden policy we previously enforced:

kubectl apply -n my-namespace -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment2
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: template-nginx
  template:
    metadata:
      labels:
        app: template-nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        securityContext:
          runAsNonRoot: true
          runAsUser: 7000
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - "ALL"
          seccompProfile:
            type: "RuntimeDefault"
        ports:
        - containerPort: 80
EOF

The rejection happens because the runAsUser value is set to 7000, which is outside of the ranges allowed by the policy:

kubectl get deploy -n my-namespace nginx-deployment -o json | jq ".status.conditions[] | select(.reason == \"FailedCreate\")"
{
  "lastTransitionTime": "2022-10-28T19:22:04Z",
  "lastUpdateTime": "2022-10-28T19:22:04Z",
  "message": "admission webhook \"clusterwide-user-group-psp.kubewarden.admission\" denied the request: User ID outside defined ranges",
  "reason": "FailedCreate",
  "status": "True",
  "type": "ReplicaFailure"
}

Summary

Pod Security Admission provide an easy way to secure Kubernetes clusters. The main goal of PSA is simplicity, because of that they do not offer all the power and flexibility of the previous Pod Security Policies.

Kubewarden can be used together with Pod Security Admission to fill this gap.

Using Pod Security Admission with Kubewarden

Example​

Summary​

Example

Summary