Export telemetry to an OpenTelemetry collector
Kubewarden Runtime Enforcer can export its violation events and one metric with OTLP (OpenTelemetry Protocol). The export is off by default.
The Helm chart does not deploy an OpenTelemetry collector. You deploy the collector, and you give its address to the chart.
|
Runtime Enforcer shows violations without telemetry. The status of each
|
What Runtime Enforcer exports
The agent and the controller export telemetry. The debugger does not.
| Component | Signal | Name | When |
|---|---|---|---|
Agent |
Event |
|
At each violation, in |
Controller |
Event |
|
When you acknowledge a violation. |
Controller |
Metric (gauge) |
|
On each status sync. The value
is the |
Runtime Enforcer sends each event as an OpenTelemetry log record. The body of the record is the name of the event.
Event attributes
Each policy_violation event has these attributes:
| Attribute | Value |
|---|---|
|
The name of the |
|
The namespace of the policy and of the pod. |
|
The name of the pod. |
|
The name of the container. |
|
The path of the executable. |
|
|
|
The name of the node. |
|
|
A policy_violation_acknowledged event has the same attributes, and
these:
| Attribute | Value |
|---|---|
|
The |
|
The reason that you gave for the acknowledgement. |
|
The time of the most recent occurrence of the violation. |
|
The name of the workload. |
|
The kind of the workload, for example |
The runtime_enforcer_active_violations metric has the policy.name and
k8s.namespace.name attributes.
Prerequisites
You need an OpenTelemetry collector with an OTLP receiver for gRPC. The controller pods and the agent pods must be able to connect to this receiver. See the OpenTelemetry documentation for how to install a collector.
Turn on the export
-
In your Helm values file, set the
global.otel.endpointvalue to the URL of the receiver:global: otel: endpoint: https://otel-collector.observability.svc.cluster.local:4317 -
Apply the values file:
helm upgrade --install runtime-enforcer kubewarden/runtime-enforcer \ --namespace runtime-enforcer \ -f values.yaml \ --wait
The scheme of the URL selects the transport:
-
https://uses TLS. -
http://uses no TLS.
The URL must start with one of these two schemes. If it does not, the controller and the agent do not start.
A change to a global.otel value restarts the controller pods and the
agent pods.
Use TLS with a private certificate authority
With an https:// endpoint, the controller and the agent trust only
public certificate authorities (CAs). If a private CA signed the
certificate of the collector, give this CA to the chart.
-
Create a Secret with the CA certificate, in PEM format, under the
ca.crtkey. Create it in the namespace of Runtime Enforcer:kubectl create secret generic otel-collector-ca \ --namespace runtime-enforcer \ --from-file=ca.crt=./ca.crt -
Set the
global.otel.caSecretNamevalue to the name of the Secret:global: otel: endpoint: https://otel-collector.observability.svc.cluster.local:4317 caSecretName: otel-collector-ca
Use mutual TLS
If the collector requires a client certificate, give the certificate to the chart.
-
Create a TLS Secret with the client certificate and its key. The Secret must have the
tls.crtandtls.keykeys:kubectl create secret tls otel-collector-client-tls \ --namespace runtime-enforcer \ --cert=./client.crt \ --key=./client.key -
Set the
global.otel.clientCertificateSecretNamevalue to the name of the Secret:global: otel: endpoint: https://otel-collector.observability.svc.cluster.local:4317 caSecretName: otel-collector-ca clientCertificateSecretName: otel-collector-client-tls
Where to set the values
The chart reads the endpoint, caSecretName, and
clientCertificateSecretName keys from four places. For each key, the
chart uses the first place that has a value:
-
The top-level
otelblock. -
The
controller.otelblock or theagent.otelblock. -
The
common.otelblock. -
The
global.otelblock.
Use global.otel for a usual installation. A parent chart can also set
the global values. If a parent chart sets global.otel and this chart
needs a different value, use the top-level otel block.
Make sure that the export works
The controller and the agent write a log line when the export is on. Look for this line in the controller log:
kubectl logs -n runtime-enforcer deployment/runtime-enforcer-controller-manager \
| grep "OTLP telemetry enabled"
Then cause a violation, as the
Quick Start shows. The
policy_violation event arrives at the collector in a few seconds.
Count the violations in the collector
Runtime Enforcer exports no counter of violations. The collector can
count the events and export the result as a Prometheus counter. This
configuration needs the count connector, the deltatocumulative
processor, and the
prometheus exporter. The otel/opentelemetry-collector-contrib image
has all three.
This collector configuration counts the two events. It also writes each
event to the log of the collector, with the debug exporter:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
connectors:
count:
logs:
runtime_enforcer_violations:
description: "Total policy violations detected by runtime-enforcer"
conditions:
- 'body == "policy_violation"'
attributes:
- key: policy.name
- key: k8s.namespace.name
- key: action
- key: node.name
runtime_enforcer_acknowledge:
description: "Total acknowledged violations by runtime-enforcer"
conditions:
- 'body == "policy_violation_acknowledged"'
attributes:
- key: policy.name
- key: k8s.namespace.name
- key: action
- key: node.name
processors:
deltatocumulative: {}
exporters:
debug:
verbosity: normal
prometheus:
endpoint: 0.0.0.0:9090
service:
pipelines:
logs:
receivers: [otlp]
exporters: [count, debug]
metrics:
receivers: [count, otlp]
processors: [deltatocumulative]
exporters: [prometheus]
This receiver uses no TLS, so the endpoint for the chart starts with
http://. For TLS, add a tls block to the receiver. See the
collector
configuration documentation.
The collector then serves the metrics on port 9090. After a violation,
the runtime_enforcer_violations_total counter looks like this:
runtime_enforcer_violations_total{action="monitor",k8s_namespace_name="default",node_name="node-1",policy_name="deploy-opensuse-deployment"} 1
The same endpoint serves the runtime_enforcer_acknowledge_total counter
and the runtime_enforcer_active_violations gauge.