This is unreleased documentation for Admission Controller 1.39-dev.

Verifying Kubewarden Admission Controller

Repository and image URIs used in this documentation page changed in v1.36. If validating previous versions, refer to the documentation page for that version.

The Kubewarden Admission Controller stack provides different attestations and assurances:

  • Provenance attestations: Inform of the build process, build dependencies, and aids in replicating the builds. The container images reach SLSA Build Level 3. See Build provenance (SLSA Build L3).

  • SBOMs attestations: Contain the software dependencies. Help downstream consumers ascertain vulnerabilities of Admission Controller and its dependencies.

  • Signed artifacts: Indicate if an artifact is authentic or not, providing supply chain security. This includes the deliverables, but also the provenance and SBOM attestations.

Admission Controller artifacts, provenance attestations and SBOMs are signed using Sigstore, with the keyless workflow. This means that the signing certificate contains the following info, where * matches any following characters:

  • issuer: https://token.actions.githubusercontent.com

  • subject: https://github.com/kubewarden/*

  • x509 certificate extension for GHA, "github_workflow_repository": kubewarden/*

Some examples in this tutorial use wildcard patterns (*) in the --certificate-identity-regexp cosign flag for simplicity. Broader patterns like https://github.com/kubewarden/* will match any workflow in any repository under the kubewarden organization, which may be less restrictive than desired.

For a more secure verification, anchor the pattern to the full path of the workflow that built the artifact. The container images, the SBOM files and the provenance files are built by one shared reusable workflow in the kubewarden/github-actions repository. Its identity is:

https://github.com/kubewarden/github-actions/.github/workflows/container-release.yml@<ref>

The <ref> is the commit of the reusable workflow, not the Admission Controller tag. For that reason the commands below use --certificate-identity-regexp with the full path and no ref. The anchored pattern accepts only an artifact that container-release.yml built.

That identity names the builder, not the release. The same builder signs the artifacts of every Kubewarden component. So each command also pins the caller repository and the release tag, with --certificate-github-workflow-repository and --certificate-github-workflow-ref. A signed file from another repository or another release does not pass. The flags do not bind the component. Every component of one release shares these certificate values, so a blob and its bundle from one component verify under another component’s filename. To check which image a file describes, read the subject field inside the file.

Container images

To verify the keyless-signed container images produced by the Admission Controller team, you can use the cosign CLI tool. For example, to verify the kubewarden/adm-controller/policy-server image, you can execute the following command:

cosign verify ghcr.io/kubewarden/adm-controller/policy-server:v1.38.2 \
  --certificate-identity-regexp '^https://github\.com/kubewarden/github-actions/\.github/workflows/container-release\.yml@' \
  --certificate-github-workflow-repository kubewarden/adm-controller \
  --certificate-github-workflow-ref 'refs/tags/v1.38.2' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Verification for ghcr.io/kubewarden/adm-controller/policy-server:v1.38.2 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

<snipped json>

You can then verify that the certificate in the returned JSON contains the correct issuer, subject, and github_workflow_repository extensions.

slsactl can also verify the images, once its identity table lists the shared workflow. At the time of writing it still expects the previous per-repository identity.

The same applies to all other images produced by the Admission Controller team, such as kubewarden/adm-controller and kubewarden/adm-controller/audit-scanner.

All container images also have SBOM and provenance files that can be used to ensure the secure supply chain of the images. You can find attestation files on the release page of the component or attached to the container image in the OCI registry.

We use Docker to build the images and their attestations. However, the cosign command does not yet support the verification of the attestations generated by Docker from the OCI registry. For this reason, you need to download the files from the release page or the registry and verify them locally. If you want to download the attestation files from the OCI registry, you can follow the Docker documentation and use tools like crane or docker itself to download the files from the registry.

When downloading the attestation files from the release page of the Admission Controller components, check them:

$ cosign verify-blob \
    --bundle policy-server-attestation-amd64-provenance.intoto.jsonl.bundle.sigstore \
    --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
    --certificate-identity-regexp='^https://github\.com/kubewarden/github-actions/\.github/workflows/container-release\.yml@' \
    --certificate-github-workflow-repository=kubewarden/adm-controller \
    --certificate-github-workflow-ref='refs/tags/v1.38.2' \
    policy-server-attestation-amd64-provenance.intoto.jsonl
Verified OK

Now that the files integrity is verified, you can inspect the SBOM and Provenance files. You can get these from the container image, using `slsactl. For example:

slsactl download provenance ghcr.io/kubewarden/adm-controller/policy-server:v1.38.2
slsactl download sbom ghcr.io/kubewarden/adm-controller/policy-server:v1.38.2

Build provenance (SLSA Build L3)

The shared workflow runs actions/attest on the multi-architecture index of each image. GitHub writes the provenance from the OIDC token of the workflow job and signs it. No build step can change it. This attestation is what gives the images SLSA Build L3.

GitHub stores the attestation in the Admission Controller repository and next to the image in the registry. Verify it with the GitHub CLI:

gh attestation verify oci://ghcr.io/kubewarden/adm-controller/policy-server:v1.38.2 \
  --repo kubewarden/adm-controller \
  --signer-workflow kubewarden/github-actions/.github/workflows/container-release.yml

Loaded digest sha256:... for oci://ghcr.io/kubewarden/adm-controller/policy-server:v1.38.2
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

--repo names the repository that stores the attestation. --signer-workflow names the workflow that signed it. The check fails for an attestation signed by any other workflow.

The SBOM and provenance files on the release page come from the container build. They carry the layer and build-arg detail that actions/attest does not. The actions/attest attestation is the one that carries the SLSA level.

Helm chart

You can find the Admission Controller Helm chart in our https:// traditional Helm repository under https://charts.kubewarden.io.

The same Helm chart is signed via Sigstore’s keyless signing, and pushed to an OCI repository that holds both the Helm chart and its provenance attestation as OCI artifacts.

The Helm chart is published alongside a Hauler manifest that lists all needed artifacts.

Verifying with Hauler

Hauler automatically verifies the artifacts on download. For more information, refer to our documentation.

Verifying manually

Since Helm 3.8.0, Helm has support for OCI registries, but because of constraints in them, they can’t be searched via helm. You can find the list of charts in GitHub Container Registry.

The kubewarden/helm-charts repository builds, signs and attests the chart with its own chart-release.yml reusable workflow. The workflow packages the chart from the commit that the signature and the attestation name, so no step outside the workflow can change it.

To verify a Helm chart, you need cosign installed. The chart carries two signature formats. Run both commands, for example:

cosign verify ghcr.io/kubewarden/charts/admission-controller:6.1.0 \
  --new-bundle-format=false \
  --certificate-identity-regexp '^https://github\.com/kubewarden/helm-charts/\.github/workflows/chart-release\.yml@' \
  --certificate-github-workflow-repository kubewarden/helm-charts \
  --certificate-github-workflow-ref 'refs/heads/main' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Verification for ghcr.io/kubewarden/charts/admission-controller:6.1.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

<snipped json>
cosign verify-attestation ghcr.io/kubewarden/charts/admission-controller:6.1.0 \
  --new-bundle-format=true \
  --type https://sigstore.dev/cosign/sign/v1 \
  --certificate-identity-regexp '^https://github\.com/kubewarden/helm-charts/\.github/workflows/chart-release\.yml@' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

You can then verify that the cert in the returned json contains the correct issuer, subject, and github_workflow_repository extensions.

The workflow also runs actions/attest on the chart. That attestation gives the chart SLSA Build L3. GitHub stores it in the kubewarden/helm-charts repository. The GitHub CLI verifies it:

gh attestation verify oci://ghcr.io/kubewarden/charts/admission-controller:6.1.0 \
  --repo kubewarden/helm-charts \
  --signer-workflow kubewarden/helm-charts/.github/workflows/chart-release.yml

The workflow also attests the packaged .tgz file that the GitHub release carries. Download the file from the release page, then verify it:

gh attestation verify admission-controller-6.1.0.tgz \
  --repo kubewarden/helm-charts \
  --signer-workflow kubewarden/helm-charts/.github/workflows/chart-release.yml

Admission Controller also follows the usual practices with regards to Helm charts. Hence, one can also find all the images in the Helm chart with a plugin such as helm-image, or with the following script:

#!/usr/bin/env bash
helm pull --untar kubewarden/kubewarden-controller && \
helm pull --untar kubewarden/kubewarden-defaults && \
{ helm template ./kubewarden-controller; helm template ./kubewarden-defaults } \
    | yq '..|.image? | select(.)' \
    | sort -u | sed 's/"//g'

which gives us:

ghcr.io/kubewarden/adm-controller/controller:v1.38.2
ghcr.io/kubewarden/adm-controller/policy-server:v1.38.2
ghcr.io/kubewarden/adm-controller/audit-scanner:v1.38.2
ghcr.io/rancher/kuberlr-kubectl:v8.0.0

Now, for each image in that list you can verify their Sigstore signatures following the instructions from the previous section.

kwctl

kwctl binaries are signed using Sigstore’s blob signing.

When you download a kwctl release each zip file contains a Sigstore bundle file that you can use for verification: kwctl-*bundle.sigstore.

In order to verify kwctl you need cosign installed, and then execute the following command:

cosign verify-blob \
  --bundle kwctl-linux-x86_64.bundle.sigstore \
  --certificate-identity-regexp 'https://github.com/kubewarden/adm-controller/.github/workflows/build-kwctl.ya?ml@refs/tags/v*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  kwctl-linux-x86_64

Verified OK

You can then verify that the cert in the returned json contains the correct issuer, subject, and github_workflow_repository extensions.

The SBOMs are signed and published in the GitHub Releases of the project.

The provenance attestation for kwctl is verified by using gh attestation verify. For example

$ gh attestation verify kwctl-linux-x86_64 --repo kubewarden/adm-controller
Loaded digest sha256:b22755eeac497614b8ab6d182c2337a5de7c74080297a29b1d68434883bb36ff for file://kwctl-linux-x86_64
Loaded 2 attestations from GitHub API
✓ Verification succeeded!
sha256:b22755eeac497614b8ab6d182c2337a5de7c74080297a29b1d68434883bb36ff was attested by:
REPO                       PREDICATE_TYPE                  WORKFLOW
kubewarden/adm-controller  https://slsa.dev/provenance/v1  .github/workflows/build-kwctl.yml@refs/tags/v1.38.2
kubewarden/adm-controller  https://slsa.dev/provenance/v1  .github/workflows/build-kwctl.yml@refs/heads/main

Policies

Policies maintained by the Admission Controller team are also signed using the Sigstore project. Similar to usual container images, one can verify them using cosign:

cosign verify ghcr.io/kubewarden/policies/verify-image-signatures:v0.2.5 \
  --certificate-identity-regexp 'https://github.com/kubewarden/*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Verification for ghcr.io/kubewarden/policies/verify-image-signatures:v0.2.5 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

  <snipped json>

You can then verify that the cert in the returned json contains the correct issuer, subject, and github_workflow_repository extensions.