This is unreleased documentation for Admission Controller 1.40-dev.

Certificate rotation

Kubewarden Admission Controller v1.17.0 removed the cert-manager dependency. Since then, the Admission Controller controller manages the certificates of the stack. It rotates each certificate before it expires, and it keeps the CA bundle of each webhook configuration correct. The CA bundle is the list of CA certificates that the API server trusts for a webhook.

Since v1.39, the controller also creates the first certificates. The Helm chart does not create a certificate, a CA bundle, or a webhook configuration of the controller. No Helm value configures these certificates. All the certificates use ECDSA P-256 keys.

If a certificate, the root CA, or a webhook configuration of the controller has a problem, refer to Recover from certificate problems.

The certificates and their secrets

The stack uses one root CA. The root CA is the certificate authority that signs all the other certificates. Each certificate is in a secret in the namespace of the Admission Controller stack, for example kubewarden.

Each controller pod has an init container. An init container runs to its end before the other containers of the pod start.

mTLS (mutual TLS) is a configuration in which the client also presents a certificate to the server.

Certificate Secret Keys Created by Valid for Used by

Root CA

kubewarden-ca

ca.crt, ca.key, and old-ca.crt after a rotation

Init container of the controller

10 years

The API server and the audit scanner use it to accept the server certificates. With mTLS, the policy servers use it to accept the client certificate of the audit scanner.

Server certificate of the controller

kubewarden-webhook-server-cert

tls.crt, tls.key

Init container of the controller

1 year

The webhook server of the controller. The API server calls this server to check and change the Admission Controller resources, for example ClusterAdmissionPolicy and PolicyServer.

Server certificate of a policy server

policy-server-<name>

tls.crt, tls.key

Controller, when you create the PolicyServer

1 year

The policy server. The API server and the audit scanner call this server to evaluate the policies.

Client certificate of the audit scanner

kubewarden-audit-scanner-client-cert

tls.crt, tls.key

Init container of the controller

1 year

The audit scanner. The policy servers require it only when mTLS is enabled.

The init container of the controller runs the controller binary with --mode=init, and it does these steps:

  1. If the secret kubewarden-ca, kubewarden-webhook-server-cert, or kubewarden-audit-scanner-client-cert is missing, it creates it. It does not change a secret that exists.

  2. It creates the two webhook configurations of the controller, or updates them to the version of the controller.

  3. It waits until the kubelet copies the certificate of the webhook server into the pod. After 5 minutes, it stops with an error, and the kubelet starts it again.

On a new installation, the init container creates the secrets before the main container starts. It finds the certificate files after a few seconds. The kubelet needs about one minute to copy a change of a secret into a pod that already mounts it.

After the installation, the controller rotates all these certificates. Only the controller instance that holds the leader Lease in the namespace of the stack does this. If you delete kubewarden-webhook-server-cert or kubewarden-audit-scanner-client-cert, the controller creates the secret again within seconds. The root CA in kubewarden-ca signs the new certificate, so the deletion does not interrupt the service.

The CA bundles of the webhook configurations

The API server calls a webhook server only if it accepts the server certificate. It accepts the certificate only if a CA in the CA bundle (caBundle) of the webhook configuration signed it. The CA bundle contains ca.crt of kubewarden-ca. If the secret holds old-ca.crt, the CA bundle contains it too.

The controller writes the CA bundle of each webhook configuration with the label app.kubernetes.io/part-of: kubewarden. There are two groups of these webhook configurations:

  • The webhook configurations of the controller, kubewarden-controller-validating-webhook-configuration and kubewarden-controller-mutating-webhook-configuration. The init container of the controller creates them. It writes the CA bundle into each new webhook. A webhook that exists keeps its CA bundle.

  • The webhook configuration of each policy. The controller creates it and writes its CA bundle. When the controller changes the webhook configuration later, it keeps the CA bundle that is in the cluster.

Do not add the label app.kubernetes.io/part-of: kubewarden to other webhook configurations. The controller replaces their CA bundle, and an uninstall deletes them.

After the creation of a webhook configuration, the controller writes its CA bundle only when the kubewarden-ca secret changes or when the CA bundle is not correct. A GitOps tool or a manual change can cause a CA bundle that is not correct. The controller writes the correct CA bundle within seconds, and it logs the message Injected the CA bundle. The controller writes these CA bundles also while a PolicyServer rolls out new pods.

When the controller rotates a certificate

The controller checks all the certificates and all the CA bundles in one pass. A pass starts in these cases:

  • When the controller starts.

  • Every 12 hours.

  • When a secret with the label app.kubernetes.io/part-of: kubewarden in the namespace of the stack is created, is deleted, or gets new data.

  • When a webhook configuration with this label is created, or its CA bundle changes.

  • After a pass that failed. The first new try starts after 5 milliseconds. The time doubles after each failure, up to about 16 minutes. For example, a pass fails when the API server is not available.

A pass does these steps in this order:

  1. If the kubewarden-ca secret is missing, the controller creates a new root CA.

  2. If the root CA expires in less than 60 days, the controller rotates the root CA. It writes a new root CA to ca.crt and ca.key, and it moves the previous root CA to old-ca.crt.

  3. If old-ca.crt is expired, the controller removes it from kubewarden-ca.

  4. The controller writes the CA bundle into each webhook configuration with the label app.kubernetes.io/part-of: kubewarden, if the CA bundle is not correct. If the controller cannot write a CA bundle, the pass stops. Then the controller does not sign a certificate in this pass.

  5. If kubewarden-webhook-server-cert or kubewarden-audit-scanner-client-cert is missing, the controller creates it.

  6. The controller rotates each server certificate and the client certificate of the audit scanner, if one of the conditions that follow is true.

The controller rotates a certificate if one of these conditions is true:

  • The certificate expires in less than 60 days.

  • The certificate is already expired.

  • The current root CA did not sign the certificate.

  • The DNS name of a server certificate is not the name of the service of the server.

  • The certificate and its key are not a valid key pair.

Thus a certificate that expired while the cluster was stopped gets a replacement at the next start of the controller.

After a rotation of the root CA, each CA bundle contains both root CAs until the previous root CA expires, up to 60 days later. Thus the API server accepts a certificate that the previous root CA signed and a certificate that the new root CA signed. The components can load their new certificates without an interruption.

The order of the steps prevents downtime for the admission requests. The controller writes the CA bundles before it signs a certificate. A server uses a new certificate only after the kubelet copies it into the pod, about one minute later. In this time, the API server reads the new CA bundle. Thus no server uses a certificate before its webhook configuration trusts the root CA that signed it.

The audit scanner trusts only the current root CA. For more information, refer to The audit scanner trusts only the current root CA.

How the components load a new certificate

No component restarts to load a new certificate.

Component How it loads a new certificate

Controller

The webhook server watches the files of its mounted secret. It loads a new certificate after the kubelet updates these files, usually within about one minute.

Policy server

The policy server watches the directories of its mounted secrets and ConfigMap. It loads a new server certificate. With mTLS, it also loads a new root CA from kubewarden-ca and a new client CA from the ConfigMap. It does this after the kubelet updates the files, usually within about one minute.

Audit scanner

Each run of the audit-scanner CronJob starts a new pod. The pod reads the current certificates.

API server

The API server uses the current CA bundle of each webhook configuration. A new CA bundle has an effect a short time after the change.

History

Before v1.39, the Helm chart contained the three secrets of the controller and the two webhook configurations of the controller. Since v1.39, the init container of the controller creates them. The chart has a pre-upgrade hook that protects these objects during the upgrade to v1.39. For the upgrade steps and the risk of helm upgrade --no-hooks, refer to v1.38 to v1.39.

Uninstall

The chart has a pre-delete hook. On helm uninstall, the hook runs the controller binary with --mode=cleanup. It deletes the webhook configurations with the label app.kubernetes.io/part-of: kubewarden, the resources of the policy servers, and the three secrets of the controller. Thus a new installation creates a new root CA.

Argo CD before version 3.3 does not run the pre-delete hook. For the objects that you must then delete, refer to Uninstall with Argo CD.

Limits

The audit scanner trusts only the current root CA

The audit scanner gets only ca.crt of kubewarden-ca, not old-ca.crt. After a rotation or a deletion of the root CA, the policy servers use their previous certificates for about one minute. In this time, a new run of the audit scanner does not accept these certificates, and its requests to these policy servers fail. The next run of the audit scanner corrects the policy reports. Admission requests are not affected.

With mTLS, the policy servers trust ca.crt of kubewarden-ca, but not old-ca.crt. Thus a policy server that loaded the new root CA rejects a client certificate that the previous root CA signed. If a run of the audit scanner is in progress during the rotation, some of its requests can fail.

Policies in the namespace of the stack

A policy with failurePolicy: Fail that covers the namespace of the Admission Controller stack can stop the rotation.

By default, a ClusterAdmissionPolicy or a ClusterAdmissionPolicyGroup does not cover this namespace. The setting allowInsideAdmissionControllerNamespace: true removes this protection. An AdmissionPolicy or an AdmissionPolicyGroup in this namespace always covers it.

If the certificate of its policy server expired, such a policy rejects each request that it matches. The policy does not run, because the API server cannot call the policy server. For this reason, the KUBEWARDEN_ALWAYS_ACCEPT_ADMISSION_REVIEWS_ON_NAMESPACE environment variable has no effect in this case.

  • If the policy matches Secret resources, the controller cannot write the new certificates.

  • If the policy matches Lease resources, the controller cannot keep its leader Lease. Then it does not check the certificates.