It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking

A routine update to a critical metrics query service caused intermittent errors and increased latency. Although logs initially pointed to DNS failures, the investigation revealed a deeper networking problem involving dropped packets and saturated AWS VPC connection tracking. The incident highlighted how Kubernetes, Cilium, AWS networking, and DNS behavior can interact in ways that obscure the true cause.

Initial Symptoms and Apparent DNS Failures

• Errors increased whenever the metrics query service was rolled out.
• The service retrieves data from metric stores for dashboards and monitor evaluations.
• Automatic retries reduced user-facing failures but increased latency.
• Service logs showed DNS errors when connecting to dependencies inside Kubernetes.
• The investigation therefore began with the cluster’s DNS infrastructure.

NodeLocal DNSCache Reaches Its Limits

• NodeLocal DNSCache runs as a node-local-dns DaemonSet on every Kubernetes node.
• DNS pods had:
  • A 64 MB memory limit
  • A max_concurrent limit of 1,000 requests
• The pods experienced out-of-memory errors and rejected requests during rollouts.
• Increasing memory to 256 MB stopped the OOM errors, but DNS failures continued.
• Request volume was far below the expected capacity:
  • Normally about 400 queries per second
  • Nearly 2,000 queries per second during rollouts
  • Expected capacity of at least 200,000 queries per second
• Upstream resolvers were marked unhealthy, suggesting that NodeLocal DNSCache could not establish or maintain connections.
• Because upstream requests could wait up to five seconds, connection failures consumed concurrency slots and made the cache appear overloaded.

Evidence of a Network Problem

• The instances were below their 5-Gbps sustained throughput limits.
• TCP retransmits increased in correlation with service rollouts.
• Engineers suspected brief traffic spikes, or microbursts, that were not visible in aggregate throughput metrics.
• This shifted the investigation from DNS configuration toward lower-level AWS networking behavior.

AWS VPC Connection Tracking

• ENA metrics revealed a significant increase in conntrack_allowance_exceeded.
• This metric counts packets dropped when VPC connection tracking becomes saturated.
• Connection tracking maintains state for network flows and supports features such as stateful EC2 security groups.
• The infrastructure used two tracking layers:
  • VPC conntrack maintained at the hypervisor level
  • Linux conntrack inside each instance
• VPC conntrack appeared saturated even though Linux conntrack contained fewer than 60,000 entries—well within the observed capacity of similar instances.
• AWS Support confirmed that conntrack capacity varies by instance type and that VPC conntrack limits could differ substantially from Linux conntrack limits.
• Scaling to larger instances resolved the symptoms, but the engineers wanted to understand the traffic pattern and find a more efficient long-term solution.

VPC Flow Logs as the Next Investigation Tool

• The team turned to Amazon VPC Flow Logs to examine the service’s low-level network behavior.
• These logs were expected to clarify why connection tracking filled up and how rollout traffic contributed to the saturation.
• The investigation was still ongoing at the point where the provided article excerpt ends.