This is unreleased documentation for Runtime Enforcer 0.12-dev.

Export telemetry to an OpenTelemetry collector

Kubewarden Runtime Enforcer can export its violation events and one metric with OTLP (OpenTelemetry Protocol). The export is off by default.

The Helm chart does not deploy an OpenTelemetry collector. You deploy the collector, and you give its address to the chart.

Runtime Enforcer shows violations without telemetry. The status of each WorkloadPolicy holds the most recent violation records. See WorkloadPolicy (monitor mode).

What Runtime Enforcer exports

The agent and the controller export telemetry. The debugger does not.

Component Signal Name When

Agent

Event

policy_violation

At each violation, in monitor mode and in protect mode.

Controller

Event

policy_violation_acknowledged

When you acknowledge a violation.

Controller

Metric (gauge)

runtime_enforcer_active_violations

On each status sync. The value is the activeViolationCount of each policy.

Runtime Enforcer sends each event as an OpenTelemetry log record. The body of the record is the name of the event.

Event attributes

Each policy_violation event has these attributes:

Attribute Value

policy.name

The name of the WorkloadPolicy.

k8s.namespace.name

The namespace of the policy and of the pod.

k8s.pod.name

The name of the pod.

container.name

The name of the container.

proc.exepath

The path of the executable.

proc.foreign_root

true when the process ran across a mount namespace boundary.

node.name

The name of the node.

action

monitor or protect.

A policy_violation_acknowledged event has the same attributes, and these:

Attribute Value

id

The id of the violation record.

reason

The reason that you gave for the acknowledgement.

lastObservedTimestamp

The time of the most recent occurrence of the violation.

workload.name

The name of the workload.

workload.kind

The kind of the workload, for example Deployment.

The runtime_enforcer_active_violations metric has the policy.name and k8s.namespace.name attributes.

Prerequisites

You need an OpenTelemetry collector with an OTLP receiver for gRPC. The controller pods and the agent pods must be able to connect to this receiver. See the OpenTelemetry documentation for how to install a collector.

Turn on the export

  1. In your Helm values file, set the global.otel.endpoint value to the URL of the receiver:

    global:
      otel:
        endpoint: https://otel-collector.observability.svc.cluster.local:4317
  2. Apply the values file:

    helm upgrade --install runtime-enforcer kubewarden/runtime-enforcer \
      --namespace runtime-enforcer \
      -f values.yaml \
      --wait

The scheme of the URL selects the transport:

  • https:// uses TLS.

  • http:// uses no TLS.

The URL must start with one of these two schemes. If it does not, the controller and the agent do not start.

A change to a global.otel value restarts the controller pods and the agent pods.

Use TLS with a private certificate authority

With an https:// endpoint, the controller and the agent trust only public certificate authorities (CAs). If a private CA signed the certificate of the collector, give this CA to the chart.

  1. Create a Secret with the CA certificate, in PEM format, under the ca.crt key. Create it in the namespace of Runtime Enforcer:

    kubectl create secret generic otel-collector-ca \
      --namespace runtime-enforcer \
      --from-file=ca.crt=./ca.crt
  2. Set the global.otel.caSecretName value to the name of the Secret:

    global:
      otel:
        endpoint: https://otel-collector.observability.svc.cluster.local:4317
        caSecretName: otel-collector-ca

Use mutual TLS

If the collector requires a client certificate, give the certificate to the chart.

  1. Create a TLS Secret with the client certificate and its key. The Secret must have the tls.crt and tls.key keys:

    kubectl create secret tls otel-collector-client-tls \
      --namespace runtime-enforcer \
      --cert=./client.crt \
      --key=./client.key
  2. Set the global.otel.clientCertificateSecretName value to the name of the Secret:

    global:
      otel:
        endpoint: https://otel-collector.observability.svc.cluster.local:4317
        caSecretName: otel-collector-ca
        clientCertificateSecretName: otel-collector-client-tls

Where to set the values

The chart reads the endpoint, caSecretName, and clientCertificateSecretName keys from four places. For each key, the chart uses the first place that has a value:

  1. The top-level otel block.

  2. The controller.otel block or the agent.otel block.

  3. The common.otel block.

  4. The global.otel block.

Use global.otel for a usual installation. A parent chart can also set the global values. If a parent chart sets global.otel and this chart needs a different value, use the top-level otel block.

Make sure that the export works

The controller and the agent write a log line when the export is on. Look for this line in the controller log:

kubectl logs -n runtime-enforcer deployment/runtime-enforcer-controller-manager \
  | grep "OTLP telemetry enabled"

Then cause a violation, as the Quick Start shows. The policy_violation event arrives at the collector in a few seconds.

Count the violations in the collector

Runtime Enforcer exports no counter of violations. The collector can count the events and export the result as a Prometheus counter. This configuration needs the count connector, the deltatocumulative processor, and the prometheus exporter. The otel/opentelemetry-collector-contrib image has all three.

This collector configuration counts the two events. It also writes each event to the log of the collector, with the debug exporter:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
connectors:
  count:
    logs:
      runtime_enforcer_violations:
        description: "Total policy violations detected by runtime-enforcer"
        conditions:
          - 'body == "policy_violation"'
        attributes:
          - key: policy.name
          - key: k8s.namespace.name
          - key: action
          - key: node.name
      runtime_enforcer_acknowledge:
        description: "Total acknowledged violations by runtime-enforcer"
        conditions:
          - 'body == "policy_violation_acknowledged"'
        attributes:
          - key: policy.name
          - key: k8s.namespace.name
          - key: action
          - key: node.name
processors:
  deltatocumulative: {}
exporters:
  debug:
    verbosity: normal
  prometheus:
    endpoint: 0.0.0.0:9090
service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [count, debug]
    metrics:
      receivers: [count, otlp]
      processors: [deltatocumulative]
      exporters: [prometheus]

This receiver uses no TLS, so the endpoint for the chart starts with http://. For TLS, add a tls block to the receiver. See the collector configuration documentation.

The collector then serves the metrics on port 9090. After a violation, the runtime_enforcer_violations_total counter looks like this:

runtime_enforcer_violations_total{action="monitor",k8s_namespace_name="default",node_name="node-1",policy_name="deploy-opensuse-deployment"} 1

The same endpoint serves the runtime_enforcer_acknowledge_total counter and the runtime_enforcer_active_violations gauge.

Turn off the export

To turn off the export, set global.otel.endpoint to an empty string. This change restarts the controller pods and the agent pods.