Kubewarden architecture
Kubewarden is a Kubernetes policy engine. It uses policies written in a programming language of your choosing. This language must generate a WebAssembly binary for Kubewarden to use.
What is a policy?​
A policy is an Open Container Initiative (OCI) artifact containing a WebAssembly module, the policy code, and the metadata required by PolicyServer performing admission request validations and mutations.
In the same manner as
Kubernetes,
Kubewarden uses the terms
'PolicyServer' when discussing the Kubewarden policy server
and
policy-server
when discussing Pod or Deployment of a Kubewarden PolicyServer.
Design principles​
Making use of core Kubernetes features​
The team designed Kubewarden to use core features of Kubernetes, without reinventing the wheel. The project utilizes a combination of:
- Kubernetes Controllers
- Custom Resource Definitions (CRDs)
- Webhooks (Validation and Mutating)
- the Control Plane's event notification system
Effectively uses Kubernetes architecture​
Kubewarden operates seamlessly within the Kubernetes ecosystem. At its core, the Kubewarden controller is a Kubernetes controller, monitoring Kubewarden Custom Resource Definitions (CRDs) and configuring Kubernetes resources to execute them. This integration ensures that Kubewarden uses the built-in Kubernetes mechanisms, such as controllers and CRDs, to watch, manage, and apply security policies efficiently.
Extensible policy definition​
Kubewarden employs CRDs to define and manage Kubewarden resources, which specify the rules for admission request validations. This design enables users to extend Kubernetes' capabilities with custom admission controls, ensuring that security and compliance policy enforcement is consistent across the cluster.
Direct admission control​
When setup by the Kubewarden controller,
the policy-server Service receives admission requests directly from the Kubernetes control plane,
using ValidationWebhooks
and MutatingWebhooks
.
This direct interaction streamlines the admission control process,
reducing latency and increasing efficiency in policy enforcement.
WebAssembly offers a sand-boxed execution environment, ensuring policies run in isolation, thus enhancing the security and stability of the policy enforcement mechanism. This isolation prevents policies from interfering with each other or with the host system, mitigating the risk of malicious code execution. WebAssembly is portable and efficient, enabling policies to run across different environments without modification. This cross-platform compatibility ensures that Kubewarden policies are versatile, and can be distributed and executed in diverse Kubernetes clusters
OCI based policy artifacts​
Policies in Kubewarden are OCI (Open Container Initiative) artifacts. This standardization makes the distribution and versioning of policies easier, Policies contain both the WebAssembly modules for enforcement logic, and metadata necessary for the PolicyServer's operation. Leveraging OCI artifacts promotes interoperability and ease of management within cloud ecosystems.
Fine-grained policy application​
Kubewarden associates policies with their own 'validation' or 'mutating' webhook, allowing for fine-grained application of admission controls. This flexibility enables administrators to tailor the enforcement of policies according to specific needs, enhancing the security and compliance posture of the Kubernetes cluster.
The Kubewarden stack​
The Kubewarden consists of these components:
-
Kubewarden Custom Resources are Kubernetes Custom Resources that simplify the process of managing policies.
Kubewarden integrates with Kubernetes using Dynamic Admission Control. In particular, Kubewarden operates as a Kubernetes Admission Webhook. The
policy-server
is the Webhook endpoint called by the Kubernetes API server to validate requests. -
The Kubewarden controller is a Kubernetes controller that reconciles Kubewarden's Custom Resources. This controller creates parts of the Kubewarden stack. It also translates Kubewarden configuration into Kubernetes directives.
The
kubewarden-controller
registers the neededMutatingWebhookConfiguration
orValidatingWebhookConfiguration
objects with the Kubernetes API server. -
Kubewarden policies are WebAssembly modules holding the validation or mutation logic. WebAssembly modules have detailed documentation in the writing policies sections.
-
The PolicyServer receives requests for validation. It validates the requests by executing Kubewarden policies.
-
The audit scanner inspects the resources already in the cluster. It identifies those violating Kubewarden policies.
Audit scanner constantly checks the resources declared in the cluster, flagging the ones that no longer adhere with the deployed Kubewarden policies.
The journey of a Kubewarden policy​
Default PolicyServer​
On a new cluster, the Kubewarden components defined are:
- Custom Resource Definitions (CRD)
- The
kubewarden-controller
Deployment - A PolicyServer Custom Resource named
default
.
When the kubewarden-controller
notices the default PolicyServer resource,
it creates a policy-server
deployment of the PolicyServer component.
Kubewarden works as a Kubernetes Admission Webhook.
Kubernetes specifies using
Transport Layer Security
(TLS) to secure all Webhook endpoints.
The kubewarden-controller
sets up this secure communication
by:
- Generating a self-signed Certificate Authority
- Use this CA to generate a TLS certificate key for the
policy-server
Service.
These objects are all stored as Secret
resources in Kubernetes.
Finally, kubewarden-controller
creates the policy-server
Deployment
and a Kubernetes ClusterIP Service
to expose it inside the cluster network.
Defining the first policy​
A policy must define which policy-server
it must run on.
It binds to a policy-server
instance.
You can have different policies with the same Wasm module and settings
running in many PolicyServers.
However, you can't have a single policy definition that runs in many PolicyServers.
The kubewarden-controller
notices the new ClusterAdmissionPolicy
resource and
so finds the bound policy-server
and reconciles it.
Reconciliation of a policy-server
​
When creating, modifying or deleting a ClusterAdmissionPolicy
or AdmissionPolicy
,
a reconciliation loop activates in kubewarden-controller
,
for the policy-server
owning the policy.
This reconciliation loop creates a ConfigMap
with all the policies bound to the policy-server
.
Then the Deployment rollout of the policy-server
starts.
It results in starting the new policy-server
instance with the updated configuration.
At start time, the policy-server
reads its configuration from the ConfigMap
and downloads all the Kubewarden policies specified.
You can download Kubewarden policies from remote HTTP servers and container registries.
You use policy settings parameters to tune a policies' behavior.
After startup and policy download the policy-server
checks the policy settings provided by the user are valid.
The policy-server
validates policy settings by invoking
the validate_setting
function exposed by each policy.
There is further documentation in the
specification reference
section of the documentation.
If one or more policies received wrong configuration parameters, from the policy specification provided by the user, then any admission requests evaluated by that policy return an error.
When Kubewarden has configured all policies,
the policy-server
spawns a pool of worker threads to evaluate incoming requests
using the Kubewarden policies specified by the user.
Finally, the policy-server
starts a HTTPS server,
listening to incoming validation requests.
Kubewarden uses the TLS key and certificate
created by the Kubewarden controller
to secure the web server.
The web server exposes each policy by a dedicated path
following the naming convention: /validate/<policy ID>
.
Making Kubernetes aware of the policy​
All policy-server
instances have a
Readiness Probe
,
that kubewarden-controller
uses to check when
the policy-server
Deployment is ready to evaluate an
AdmissionReview
.
Once Kubewarden marks the policy-server
deployment as 'uniquely reachable' or Ready
,
the kubewarden-controller
makes the Kubernetes API server aware of the new policy.
This is by creating either a MutatingWebhookConfiguration
or a ValidatingWebhookConfiguration
object.
In this context, 'uniquely reachable',
means that all the PolicyServer instances in the cluster have the latest policy configuration installed.
The distinction, is a fine point, but is necessary,
due to how roll-out of PolicyServers works.
It's possible to have the same policy,
on different PolicyServers with different configurations.
Each policy has a dedicated
MutatingWebhookConfiguration
or ValidatingWebhookConfiguration
pointing to the Webhook endpoint served by policy-server
.
The endpoint is reachable at the /validate/<policy ID>
URL.
Policy in action​
Now that all the necessary plumbing is complete,
Kubernetes starts sending Admission Review requests to the right policy-server
endpoints.
A policy-server
receives the Admission Request object and,
based on the endpoint that received the request,
uses the correct policy to evaluate it.
Kubewarden evaluates each policy inside its own dedicated WebAssembly sand-box.
The communication between a policy-server
instance (the "host")
and the WebAssembly policy (the "guest")
uses the waPC communication protocol.
The protocol description is part of the
writing policies documentation.
Policies can also use the interfaces provided by the
Web Assembly System Interface
(WASI).
How Kubewarden handles many PolicyServer and policies​
A cluster can have many PolicyServers and Kubewarden policies defined. There are benefits of having many PolicyServers:
-
You can isolate noisy namespaces or tenants, those generating many policy evaluations, from the rest of the cluster so as not to adversely affect other cluster operations.
-
You can run mission-critical policies in a dedicated PolicyServer pool, making your infrastructure more resilient.
A PolicyServer resource defines each policy-server
and a ClusterAdmissionPolicy
or AdmissionPolicy
resource defines each policy.
A ClusterAdmissionPolicy
and an AdmissionPolicy
bind to a policy-server
.
Any ClusterAdmissionPolicy
not specifying a policy-server
binds to the default PolicyServer.
If a ClusterAdmissionPolicy
references a policy-server
that doesn't exist, its state is unschedulable
.
Each policy-server
defines many validation endpoints,
one for each policy defined in its configuration file.
You can load the same policy many times,
with different configuration parameters.
The ValidatingWebhookConfiguration
and MutatingWebhookConfiguration
resources
make the Kubernetes API server aware of these policies.
Then kubewarden-controller
keeps the API server
and configuration resources in synchronization.
The Kubernetes API server dispatches incoming admission requests
to the correct validation endpoint exposed by policy-server
.