Kubewarden Runtime Enforcer use cases

I want each pod to run only the executables that its job needs. I do not know the full list for each workload. The list is different for each image, and it changes with each release.

I install Runtime Enforcer once, for the whole cluster. An agent runs on each node. It observes each process that starts inside a container. For each Deployment, StatefulSet, DaemonSet, Job, and CronJob, Runtime Enforcer creates a WorkloadPolicyProposal. The proposal lists the allowed executables for each container of the workload. Init containers and sidecars get their own lists. This list is the allow-list.

The agent records the absolute path of each executable. When a container runs a script, the agent records the path of the script. It does not record the interpreter.

Learning starts when the agent starts. Processes from before that moment are not in the proposal. If a workload is quiet, I restart it so that its start-up processes are observed. I limit learning to selected namespaces with a namespace selector.

Each command that runs during learning becomes an allowed executable. This includes commands from kubectl exec. I review the proposal before I promote it.

See Learn phase, Learning mode configuration, and Learning cap.

I do not want a wrong rule to stop my application. I promote the proposal in monitor mode first. To do this, I set the label runtimeenforcer.kubewarden.io/promote to monitor on the proposal. The kubectl plugin can set the label for me:

Runtime Enforcer creates a WorkloadPolicy and removes the proposal. In monitor mode, the agent reports each executable that the policy does not allow. It does not block the executable.

I watch the Active Violations column of the policy. For each violation, I make one of two decisions. If the executable is expected, I add it to the allow-list of the policy, under spec.rulesByContainer. The plugin command kubectl runtime-enforcer policy allow does this edit for me. The violation is cleared. If the executable is not expected, but I accept the risk, I acknowledge the violation. I add the annotation runtimeenforcer.kubewarden.io/acknowledge-<id> with a reason, or I run kubectl runtime-enforcer policy ack. The reason stays in the policy status as a record.

When the policy has no active violations for a period that I trust, I set the mode to protect. If a new release adds an executable, the rollout can fail. Then I set the mode back to monitor, add the executable, and try again.

The policy is bound to my pods with the runtimeenforcer.kubewarden.io/policy label. The same image in a different namespace can have a different policy. My policy does not affect other teams.

See Monitor phase, Protect phase, Quick mapping to CRDs, Promote a proposal, Allowing and denying executables, and Acknowledge a violation.

An attacker who gets into a container tries to run new programs: a shell, a package manager, a downloaded binary, or a scanner. None of these are in the allow-list of the workload.

In protect mode, the agent checks each process before it starts. The check runs in the kernel, in an eBPF program. If the executable is not in the allow-list, the kernel refuses the start. The process never runs. The agent records a violation with the pod, the container, the executable, and the node.

The check also covers an interactive shell that starts with kubectl exec, if the shell is not in the allow-list. It covers the entrypoint of the container, because the policy is attached before the first process starts. Short-lived pods, such as Jobs, are covered in the same way.

If a pod refers to a policy that does not exist, the agent does not start the container. This is the default behavior. A pod is not left without protection by mistake. The fail-open entry in the glossary describes the setting that changes this.

Runtime Enforcer controls which executables start. It does not inspect what an allowed executable does after it starts. Code that runs inside an allowed process, without a new executable, is not seen. I combine Runtime Enforcer with the other security tools that I run.

See Protect phase and Required plugins.

My cluster has many workloads and many policies. I need one view of the state.

The kubectl runtime-enforcer policy show protection command lists each workload with its policy, its mode, and its status. I see at once which workloads have no policy, which are in monitor mode, and which are in protect mode.

Each policy has two counters. The activeViolationCount shows the violations that need a decision. The violationCount shows all the violations that the policy has ever recorded. I sort by the active count to find the noisy policies.

The policy status holds the 100 most recent violation records. The controller updates it every 30 seconds. For a longer history and for alerts, I use the telemetry pipeline. The agent emits a policy_violation event for each violation, and a policy_violation_acknowledged event for each acknowledgement. The events go to the OpenTelemetry collector that Runtime Enforcer installs, or to my own collector. The collector exports a Prometheus counter for violations. The controller exports a gauge for active violations. I set alerts on these metrics.

See Quick overview with the kubectl plugin and Use an external collector.

I run the same workloads in many clusters. I do not want to learn a policy in each cluster.

A WorkloadPolicy is a namespaced Kubernetes resource. It contains only the mode and the allow-list for each container. It has no cluster-specific state. I learn the policy in a staging cluster and promote it. Then I export the policy and commit it to my GitOps repository. My GitOps tool applies it to each production cluster, in protect mode.

The pod must carry the policy label when it is created. I add the label to the pod template of the workload manifest, in the same repository. A policy in the Kubewarden Admission Controller can require this label on each pod in selected namespaces.

When I uninstall Runtime Enforcer, the policies stay in the cluster. When I install it again, protection resumes.

See Quick mapping to CRDs.

Runtime Enforcer does not intend to: