When default deny blocked the Kubernetes API

An undocumented dependency becomes visible

In Learning the network by denying it first, I described how I use default-deny policies to discover application data flows in my homelab. This incident put that approach into practice inside the networking platform itself.

I upgraded my MikroTik network devices, which caused a network outage in my homelab. While checking the cluster after the outage, I found that a newly created calico-apiserver replica on a worker node stayed at 0/1 and repeatedly restarted. The existing replica on control-plane-01 was still healthy.

The failing Pod could not retrieve configuration from the Kubernetes API. The relevant part of the error was:

unable to load configmap based request-header-client-ca-file:
Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/..."

The timing made routing, MTU and node connectivity my first suspects. The fix was an explicit egress permission to the control-plane endpoint. Allowing only the Service address shown in the log had not been enough.

The namespace was inside default deny

My Calico GlobalNetworkPolicy, zt-default-deny, establishes the cluster’s ingress and egress baseline. Infrastructure namespaces such as kube-system, calico-system, tigera-operator, longhorn-system and metallb-system are excluded. The calico-apiserver namespace was still in scope.

That meant its Pods needed explicit permission for outbound API traffic. I wanted to keep that boundary and describe the dependency.

There was already an operator-managed policy:

kubectl -n calico-apiserver get networkpolicy allow-apiserver -o yaml

Its relevant configuration allowed ingress on TCP/5443:

spec:
  podSelector:
    matchLabels:
      apiserver: "true"
  policyTypes:
    - Ingress
  ingress:
    - ports:
        - protocol: TCP
          port: 5443

It did not grant egress. Its owner reference pointed to the Tigera operator’s APIServer resource, so I created a separate policy for the missing outbound dependency. That keeps my exception separate from configuration the operator reconciles.

The address in the log was only the starting point

My first attempt allowed 10.96.0.1/32 on TCP/443. That matched the application error, but the replica still failed.

I then inspected the backend of the Kubernetes Service:

kubectl -n default get endpoints kubernetes -o wide

The result was:

NAME         ENDPOINTS
kubernetes   172.16.20.200:6443

The current API to inspect this is EndpointSlice:

kubectl -n default get endpointslice \
  -l kubernetes.io/service-name=kubernetes -o wide

The Service provides the address the Pod connects to. Service translation forwards that connection to the backend, changing the destination IP and port.

In my homelab, 172.16.20.200 is the actual backend address, assigned to the control-plane node’s physical interface. It is also the address I use for SSH. It is not a separate Kubernetes virtual IP. That is my cluster’s configuration, not a requirement that every Kubernetes API endpoint must use a physical NIC address.

The distinction matters at the policy enforcement point. Calico documents ordinary workload policy as being evaluated after connection tracking and destination NAT. That is consistent with the recovery after I allowed the translated destination. I have not yet captured the packet and rule traversal to demonstrate the exact local dataplane path. Calico GlobalNetworkPolicy reference

Kubernetes also leaves the ordering of address rewriting relative to NetworkPolicy processing implementation dependent. A Service-IP rule should therefore be tested with the actual CNI and Service implementation. Kubernetes NetworkPolicy documentation

The working exception

I used the same two-destination pattern already present in my Flux policies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-kubernetes-api
  namespace: calico-apiserver
spec:
  podSelector:
    matchLabels:
      apiserver: "true"
  policyTypes:
    - Egress
  egress:
    # Kubernetes API Service address
    - to:
        - ipBlock:
            cidr: 10.96.0.1/32
      ports:
        - protocol: TCP
          port: 443
    # Actual Kubernetes API endpoint in my homelab
    - to:
        - ipBlock:
            cidr: 172.16.20.200/32
      ports:
        - protocol: TCP
          port: 6443

After applying this policy, the worker replica became healthy. Both replicas were running, and the new Pod could complete initialization.

The observed change was adding 172.16.20.200:6443. Keeping both entries records the Service and backend addresses used in my configuration; it does not prove both are necessary in this dataplane. Testing the backend-only rule remains a useful follow-up.

Policy order also matters. My baseline uses order 1000. A namespaced allow rule does not universally override a Calico deny: an earlier matching terminal deny can prevent the allow from being reached. Review tiers and ordering alongside the rule contents. Calico policy ordering and actions

This rule grants network reachability. Kubernetes authentication and RBAC still determine what the workload can do through the API.

Why the existing replica was misleading

The old replica had been running for approximately 194 days. Its healthy state showed that it was operating, but did not demonstrate that a newly scheduled Pod could initialize under the current policies.

The replicas also ran on different nodes. Startup dependencies, established connections and the difference between local-node and remote-node traffic are possible explanations. I did not isolate those factors, so I cannot attribute the difference to Pod age alone.

What I could establish was narrower: the new replica failed to reach the API during initialization, allowing only the ClusterIP did not resolve it, and adding the backend destination restored readiness.

The MikroTik upgrades caused the network outage. During the recovery checks, the failing replica exposed a separate, existing gap in my egress policy. The outage explains why I was investigating the cluster; the missing API permission explains why the new replica could not initialize.

A repeatable check for the next workload

The next time a workload fails under default deny, I will compare the application destination with the actual backend before expanding the rule.

# Identify the failing request and Pod placement
kubectl -n calico-apiserver get pods -o wide
kubectl -n calico-apiserver logs <failing-pod> --previous

# Inspect the Service and its backend
kubectl -n default get service kubernetes -o wide
kubectl -n default get endpointslice \
  -l kubernetes.io/service-name=kubernetes -o yaml

# Review policies, selectors and global ordering
kubectl -n calico-apiserver get networkpolicy -o yaml
kubectl get globalnetworkpolicies.crd.projectcalico.org -o yaml

After a change, I will check a fresh Pod’s initialization as well as readiness and logs. I also want a negative test showing that an unrelated destination remains blocked. Recovery proves the required path works; it does not by itself prove the remaining boundary is correct.

This is the practical value of default deny for me. An implicit platform dependency becomes an explicit, reviewable rule.

Cilium gives the roadmap a concrete test

The current exception contains the physical implementation of a logical dependency: 172.16.20.200/32. If I change the control-plane address or add API backends, I need to review it against the Service’s actual endpoints.

Cilium is already on my roadmap for FQDN-aware egress. It also provides a kube-apiserver entity for expressing API access without listing backend IPs. Its documentation recommends this entity for the special default/kubernetes Service. Cilium Layer 3 policies

For the migration lab, an illustrative policy would be:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-kubernetes-api
  namespace: api-access-lab
spec:
  endpointSelector:
    matchLabels:
      app: api-client
  egress:
    - toEntities:
        - kube-apiserver
      toPorts:
        - ports:
            - port: "6443"
              protocol: TCP

This is a future test, not a policy I have deployed. It selects a deliberately labelled test client and uses my current backend port. I will validate port matching with the chosen Cilium Service implementation.

A full CNI replacement would normally remove calico-apiserver, so the migration test should use an API client that remains relevant, such as a dedicated test workload or a Flux controller. Cilium also treats node addresses differently from ordinary CIDR destinations by default; carrying the existing ipBlock rule across unchanged is not a migration test. Cilium CIDR selection

Way forward

The goal is to make required communication understandable and reproducible. This incident gave me a specific dependency to document today and a concrete acceptance test for the next stage of the homelab.

The existing baseline and repository structure are documented in my homelab Network Policies page.