|
This is unreleased documentation for Admission Controller 1.40-dev. |
Certificate rotation
Kubewarden Admission Controller v1.17.0 removed the cert-manager dependency. Since then, the Admission Controller controller manages the certificates of the stack. It rotates each certificate before it expires, and it keeps the CA bundle of each webhook configuration correct. The CA bundle is the list of CA certificates that the API server trusts for a webhook.
Since v1.39, the controller also creates the first certificates. The Helm chart does not create a certificate, a CA bundle, or a webhook configuration of the controller. No Helm value configures these certificates. All the certificates use ECDSA P-256 keys.
If a certificate, the root CA, or a webhook configuration of the controller has a problem, refer to Recover from certificate problems.
The certificates and their secrets
The stack uses one root CA. The root CA is the certificate authority that
signs all the other certificates. Each certificate is in a secret in the
namespace of the Admission Controller stack, for example kubewarden.
Each controller pod has an init container. An init container runs to its end before the other containers of the pod start.
mTLS (mutual TLS) is a configuration in which the client also presents a certificate to the server.
| Certificate | Secret | Keys | Created by | Valid for | Used by |
|---|---|---|---|---|---|
Root CA |
|
|
Init container of the controller |
10 years |
The API server and the audit scanner use it to accept the server certificates. With mTLS, the policy servers use it to accept the client certificate of the audit scanner. |
Server certificate of the controller |
|
|
Init container of the controller |
1 year |
The webhook server of the controller. The API server calls this server to
check and change the Admission Controller resources, for example
|
Server certificate of a policy server |
|
|
Controller, when you create the |
1 year |
The policy server. The API server and the audit scanner call this server to evaluate the policies. |
Client certificate of the audit scanner |
|
|
Init container of the controller |
1 year |
The audit scanner. The policy servers require it only when mTLS is enabled. |
The init container of the controller runs the controller binary with
--mode=init, and it does these steps:
-
If the secret
kubewarden-ca,kubewarden-webhook-server-cert, orkubewarden-audit-scanner-client-certis missing, it creates it. It does not change a secret that exists. -
It creates the two webhook configurations of the controller, or updates them to the version of the controller.
-
It waits until the kubelet copies the certificate of the webhook server into the pod. After 5 minutes, it stops with an error, and the kubelet starts it again.
On a new installation, the init container creates the secrets before the main container starts. It finds the certificate files after a few seconds. The kubelet needs about one minute to copy a change of a secret into a pod that already mounts it.
After the installation, the controller rotates all these certificates. Only
the controller instance that holds the leader Lease in the namespace of the
stack does this. If you delete kubewarden-webhook-server-cert or
kubewarden-audit-scanner-client-cert, the controller creates the secret
again within seconds. The root CA in kubewarden-ca signs the new
certificate, so the deletion does not interrupt the service.
The CA bundles of the webhook configurations
The API server calls a webhook server only if it accepts the server
certificate. It accepts the certificate only if a CA in the CA bundle
(caBundle) of the webhook configuration signed it. The CA bundle contains ca.crt of kubewarden-ca. If
the secret holds old-ca.crt, the CA bundle contains it too.
The controller writes the CA bundle of each webhook configuration with the
label app.kubernetes.io/part-of: kubewarden. There are two groups of these
webhook configurations:
-
The webhook configurations of the controller,
kubewarden-controller-validating-webhook-configurationandkubewarden-controller-mutating-webhook-configuration. The init container of the controller creates them. It writes the CA bundle into each new webhook. A webhook that exists keeps its CA bundle. -
The webhook configuration of each policy. The controller creates it and writes its CA bundle. When the controller changes the webhook configuration later, it keeps the CA bundle that is in the cluster.
Do not add the label app.kubernetes.io/part-of: kubewarden to other webhook
configurations. The controller replaces their CA bundle, and an uninstall
deletes them.
After the creation of a webhook configuration, the controller writes its CA
bundle only when the kubewarden-ca secret changes or when the CA bundle is
not correct. A GitOps tool or a manual change can cause a CA bundle that is
not correct. The controller writes the correct CA bundle within seconds, and
it logs the message Injected the CA bundle. The controller writes these CA
bundles also while a PolicyServer rolls out new pods.
When the controller rotates a certificate
The controller checks all the certificates and all the CA bundles in one pass. A pass starts in these cases:
-
When the controller starts.
-
Every 12 hours.
-
When a secret with the label
app.kubernetes.io/part-of: kubewardenin the namespace of the stack is created, is deleted, or gets new data. -
When a webhook configuration with this label is created, or its CA bundle changes.
-
After a pass that failed. The first new try starts after 5 milliseconds. The time doubles after each failure, up to about 16 minutes. For example, a pass fails when the API server is not available.
A pass does these steps in this order:
-
If the
kubewarden-casecret is missing, the controller creates a new root CA. -
If the root CA expires in less than 60 days, the controller rotates the root CA. It writes a new root CA to
ca.crtandca.key, and it moves the previous root CA toold-ca.crt. -
If
old-ca.crtis expired, the controller removes it fromkubewarden-ca. -
The controller writes the CA bundle into each webhook configuration with the label
app.kubernetes.io/part-of: kubewarden, if the CA bundle is not correct. If the controller cannot write a CA bundle, the pass stops. Then the controller does not sign a certificate in this pass. -
If
kubewarden-webhook-server-certorkubewarden-audit-scanner-client-certis missing, the controller creates it. -
The controller rotates each server certificate and the client certificate of the audit scanner, if one of the conditions that follow is true.
The controller rotates a certificate if one of these conditions is true:
-
The certificate expires in less than 60 days.
-
The certificate is already expired.
-
The current root CA did not sign the certificate.
-
The DNS name of a server certificate is not the name of the service of the server.
-
The certificate and its key are not a valid key pair.
Thus a certificate that expired while the cluster was stopped gets a replacement at the next start of the controller.
After a rotation of the root CA, each CA bundle contains both root CAs until the previous root CA expires, up to 60 days later. Thus the API server accepts a certificate that the previous root CA signed and a certificate that the new root CA signed. The components can load their new certificates without an interruption.
The order of the steps prevents downtime for the admission requests. The controller writes the CA bundles before it signs a certificate. A server uses a new certificate only after the kubelet copies it into the pod, about one minute later. In this time, the API server reads the new CA bundle. Thus no server uses a certificate before its webhook configuration trusts the root CA that signed it.
The audit scanner trusts only the current root CA. For more information, refer to The audit scanner trusts only the current root CA.
How the components load a new certificate
No component restarts to load a new certificate.
| Component | How it loads a new certificate |
|---|---|
Controller |
The webhook server watches the files of its mounted secret. It loads a new certificate after the kubelet updates these files, usually within about one minute. |
Policy server |
The policy server watches the directories of its mounted secrets and
|
Audit scanner |
Each run of the |
API server |
The API server uses the current CA bundle of each webhook configuration. A new CA bundle has an effect a short time after the change. |
History
Before v1.39, the Helm chart contained the three secrets of the controller and
the two webhook configurations of the controller. Since v1.39, the init
container of the controller creates them. The chart has a pre-upgrade hook
that protects these objects during the upgrade to v1.39. For the upgrade steps
and the risk of helm upgrade --no-hooks, refer to
v1.38 to v1.39.
Uninstall
The chart has a pre-delete hook. On helm uninstall, the hook runs the
controller binary with --mode=cleanup. It deletes the webhook configurations
with the label app.kubernetes.io/part-of: kubewarden, the resources of the
policy servers, and the three secrets of the controller. Thus a new
installation creates a new root CA.
Argo CD before version 3.3 does not run the pre-delete hook. For the objects that you must then delete, refer to Uninstall with Argo CD.
Limits
The audit scanner trusts only the current root CA
The audit scanner gets only ca.crt of kubewarden-ca, not old-ca.crt.
After a rotation or a deletion of the root CA, the policy servers use their
previous certificates for about one minute. In this time, a new run of the
audit scanner does not accept these certificates, and its requests to these
policy servers fail. The next run of the audit scanner corrects the policy
reports. Admission requests are not affected.
With mTLS, the policy servers trust ca.crt of kubewarden-ca, but not
old-ca.crt. Thus a policy server that loaded the new root CA rejects
a client certificate that the previous root CA signed. If a run of the audit
scanner is in progress during the rotation, some of its requests can fail.
Policies in the namespace of the stack
A policy with failurePolicy: Fail that covers the namespace of the
Admission Controller stack can stop the rotation.
By default, a ClusterAdmissionPolicy or a ClusterAdmissionPolicyGroup
does not cover this namespace. The setting
allowInsideAdmissionControllerNamespace: true removes this protection. An
AdmissionPolicy or an AdmissionPolicyGroup in this namespace always
covers it.
If the certificate of its policy server expired, such a policy rejects each
request that it matches. The policy does not run, because the API server
cannot call the policy server. For this reason, the
KUBEWARDEN_ALWAYS_ACCEPT_ADMISSION_REVIEWS_ON_NAMESPACE environment
variable has no effect in this case.
-
If the policy matches
Secretresources, the controller cannot write the new certificates. -
If the policy matches
Leaseresources, the controller cannot keep its leaderLease. Then it does not check the certificates.
For more information, refer to Allowing evaluations in the Admission Controller namespace.