|
This is unreleased documentation for Network Enforcer 0.4-dev. |
Emergency disable
During an incident, a Network Enforcer policy can block traffic that your team needs. This page stops the blocking with two commands, and restores it with one command once the incident is over.
The commands on this page assume the release name network-enforcer
in namespace network-enforcer. Substitute the names from
helm list -A if you installed elsewhere.
What enforcement depends on
The data plane enforces a rendered policy: an Istio
AuthorizationPolicy, or a Kubernetes NetworkPolicy on Calico and
Cilium. The controller keeps rendered policies in sync with their
WorkloadNetworkPolicy resources. It recreates a rendered policy after
you delete it.
If the controller stops, every rendered policy stays. See How Kubewarden Network Enforcer scales for why the controller can stop without a gap in enforcement. Stopping the controller does not stop the blocking by itself. You must also remove the rendered policies, which the next section covers.
Stop blocking
1. Scale the controller to zero
Scale the controller Deployment to zero replicas, then wait for its pod to terminate:
kubectl scale deployment network-enforcer-controller-manager \
-n network-enforcer \
--replicas=0
kubectl wait --for=delete pod \
-l app.kubernetes.io/component=controller \
-n network-enforcer \
--timeout=60s
|
Do not go to the next step until the wait command returns. A running controller watches every rendered policy and recreates one the instant you delete it. |
While the controller is at zero, learning, promotion, status sync, and
violation events stop. The admission webhook is also unavailable, so
WorkloadNetworkPolicy creates and updates fail. Deletes of rendered
policies are not subject to this webhook. On Istio, fluent-bit keeps
running but cannot forward logs.
2. Delete the rendered policies
The controller labels every rendered policy with
app.kubernetes.io/managed-by=network-enforcer. One label selector
finds all of them, and nothing else in the cluster.
Run the command for your provider.
3. Make sure traffic flows
List the rendered policies again, with the same selector you deleted with:
kubectl get networkpolicy,authorizationpolicy -A \
-l app.kubernetes.io/managed-by=network-enforcer
The list is empty. Traffic that a rendered policy blocked now flows, within seconds of the deletion. Run the request that the incident needed, and make sure that it succeeds.
Restore protection
Scale the controller back up:
kubectl scale deployment network-enforcer-controller-manager \
-n network-enforcer \
--replicas=1
Use the replica count that you ran before the incident, if you changed
controller.replicas from its default.
On startup, the controller reconciles every WorkloadNetworkPolicy and
renders each protect policy again. You do not patch anything, and you
do not relabel anything. Run the list
command from step 3 again, and the rendered policies reappear.
Do not
Do not delete a rendered policy while the controller runs
The controller watches every rendered policy that it owns. If you delete one while the controller runs, the controller recreates it at once.
Do not patch spec.mode while the controller is down
The fail-closed admission webhook rejects the patch while the controller is unavailable. Existing rendered policies keep blocking traffic until you delete them as described in step 2.
See also
-
A mode change explains what the controller does to a rendered policy when you change
spec.mode, while the controller runs. -
Controller replicas covers the default replica count and how to run more than one.
-
Traffic still flows helps when traffic is not blocked the way that you expect, outside an incident.