--- title: "Resolve Kubernetes CoreDNS Timeouts Like a Pro" description: "Trace CoreDNS timeouts through all five Kubernetes DNS hops: ndots:5 lookups, the 5-second conntrack race, cache sizing, and NodeLocal DNSCache." canonical: "https://adamtheautomator.com/fix-coredns-timeouts-kubernetes/" --- # Resolve Kubernetes CoreDNS Timeouts Like a Pro > Trace CoreDNS timeouts through all five Kubernetes DNS hops: ndots:5 lookups, the 5-second conntrack race, cache sizing, and NodeLocal DNSCache. Source: https://adamtheautomator.com/fix-coredns-timeouts-kubernetes/ --- ATA Learning Tap to hide [ ATA Learning ](/) * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Search for: * [](https://twitter.com/adbertram) * [](https://github.com/Adam-the-Automator) * [](https://www.linkedin.com/company/adam-the-automator-llc) * [](/feed/) ![Resolve Kubernetes CoreDNS Timeouts Like a Pro](https://adamtheautomator.com/wp-content/uploads/publisher/3ec5d9c85b2b8140a365ef14a6b86f61/961d44c379185b8fc0fc222f19db075aa840e3336ede327b42a819523a3bdc25.webp) # Resolve Kubernetes CoreDNS Timeouts Like a Pro [![](https://secure.gravatar.com/avatar/d0b9d42e21e5622713f8b693aa5c0f9244d5f7dd200ed29b8398f52dee5de337?s=192&d=mm&r=g)Adam Bertram](https://adamtheautomator.com/author/adam-bertram/)19 October 202615 min. read Categories: [DevOps](/category/devops/) Tags:[Kubernetes](/tag/kubernetes/)[DNS](/tag/dns/)[Networking](/tag/networking/)[Linux](/tag/linux/) Table of Contents * [Prerequisites](#prerequisites) * [The Five Hops a Query Travels](#the-five-hops-a-query-travels) * [Start With the Endpoints Behind the ClusterIP](#start-with-the-endpoints-behind-the-clusterip) * [Read the Shape of the Failure Before You Change Anything](#read-the-shape-of-the-failure-before-you-change-anything) * [ndots:5 Turns One Lookup Into Eight](#ndots5-turns-one-lookup-into-eight) * [Lowering ndots Without Breaking Service Discovery](#lowering-ndots-without-breaking-service-discovery) * [The Exactly-Five-Second Stall Is a Kernel Race](#the-exactly-five-second-stall-is-a-kernel-race) * [The Per-Workload Fix for glibc](#the-per-workload-fix-for-glibc) * [When CoreDNS Itself Is the Problem](#when-coredns-itself-is-the-problem) * [A Corefile Worth Running](#a-corefile-worth-running) * [Size Both Cache Tables](#size-both-cache-tables) * [Size CoreDNS From Measured Load](#size-coredns-from-measured-load) * [Forward: Order, Policy, and the REFUSED You Did Not Expect](#forward-order-policy-and-the-refused-you-did-not-expect) * [autopath Is Not a Free Win](#autopath-is-not-a-free-win) * [NodeLocal DNSCache Removes the Race Entirely](#nodelocal-dnscache-removes-the-race-entirely) * [Verifying the Fix with PromQL](#verifying-the-fix-with-promql) * [The Check That Covers Hop 2](#the-check-that-covers-hop-2) * [Frequently Asked Questions](#frequently-asked-questions) * [Why Does a DNS Lookup Take Exactly 5 Seconds?](#why-does-a-dns-lookup-take-exactly-5-seconds) * [Does Lowering ndots Break StatefulSet Lookups?](#does-lowering-ndots-break-statefulset-lookups) * [Do I Need NodeLocal DNSCache in Every Cluster?](#do-i-need-nodelocal-dnscache-in-every-cluster) * [What Does SERVFAIL Mean Here?](#what-does-servfail-mean-here) * [What to Change First](#what-to-change-first) A pod stops resolving `api.stripe.com` for five seconds, then resolves it fine. The client logs `i/o timeout`, retries on its own, and the second attempt returns an address in two milliseconds. No deployment ran. No node was drained. Forty minutes later a different pod on a different node stalls at the same five second mark. A resolver that is down fails every query. A Service name that does not exist fails every time. This failure passes on retry and moves between nodes, which rules out both obvious suspects and points at the path a DNS query takes through the Linux kernel. That path has five hops, and each leaves a shape you can recognize from `kubectl` output, a conntrack counter, or the latency graph the PromQL histogram builds. This guide walks them in order, sorts the three causes behind most incidents by the evidence each leaves, and names the fix, then proves it with PromQL and a client-side timing loop. ## Prerequisites You can run the diagnosis on any cluster you can read pods and logs from. The hands-on parts need: * `kubectl` with permission to read `kube-system` and create pods in one namespace. * A probe pod in the namespace you test from. The Kubernetes DNS debugging guide ships one: `kubectl apply -f https://kubernetes.io/examples/admin/dns/dnsutils.yaml`, then confirm it with `kubectl get pods dnsutils`. * Shell access to one worker node for the conntrack commands, plus `conntrack-tools`, which minimal node images omit. * CoreDNS metrics on port 9153, which the [prometheus plugin](https://coredns.io/plugins/metrics/) exposes when the `Corefile` includes `prometheus`. ## The Five Hops a Query Travels Every lookup crosses five stages, and naming them turns a vague “DNS is broken” ticket into a list of places to look. | Hop | What lives there | What a failure looks like | | --- | --- | --- | | 1\. Stub resolver | `getaddrinfo()` in the container’s C library, reading `/etc/resolv.conf` | Wrong `ndots`, missing search domains | | 2\. ClusterIP DNAT | iptables or IPVS rules pointing the ClusterIP at a CoreDNS pod | Packets dropped before any DNS process sees them | | 3\. CoreDNS | The plugin chain in the `Corefile`: `kubernetes`, `cache`, `forward` | `SERVFAIL`, `REFUSED`, rising latency, `OOMKilled` | | 4\. Upstream | The node resolver, a VPC resolver, or a stub zone | `i/o timeout` in the CoreDNS log, `NXDOMAIN` for real names | | 5\. Return path | conntrack state for the UDP flow | An answer that arrives after the client gave up | ### Start With the Endpoints Behind the ClusterIP Hop 2 is where Kubernetes starts behaving like a kernel networking product. Start with the endpoints behind the ClusterIP: an empty list means every lookup fails at hop 2 whatever the `Corefile` says. ```bash kubectl -n kube-system get endpoints kube-dns ``` Endpoints is deprecated in favor of [EndpointSlice](https://kubernetes.io/blog/2025/04/24/endpoints-deprecation/) as of Kubernetes v1.33, so newer clusters run: ```bash kubectl -n kube-system get endpointslice -l kubernetes.io/service-name=kube-dns ``` The diagram below traces all five hops and marks where a packet dies at each one. The [Kubernetes architecture diagram](https://adamtheautomator.com/kubernetes-architecture-diagram/) covers the control plane and the worker nodes, including kube-proxy, behind those hops, so the evidence you just collected points at a stage rather than a symptom. ![Five DNS hops](https://adamtheautomator.com/wp-content/uploads/publisher/1643607b726667cd7d79759f540992de5c30f01620c0416d73f6e006e57e291c.jpg) ## Read the Shape of the Failure Before You Change Anything The timing answers the important question faster than any config diff: where does the packet stop? | What you observe | Response code at the client | CoreDNS restarts | Hop to inspect | First check | | --- | --- | --- | --- | --- | | Stalls cluster at 5.00s, retry succeeds | none, the client times out | 0 | 2, conntrack | `conntrack -S` on the node | | Latency scattered from 200ms to 4s, worse at high QPS | `NOERROR`, slow | 0 | 3, CoreDNS capacity | request duration histogram | | Fast `SERVFAIL` for external names only | `SERVFAIL` | 0 | 4, upstream | CoreDNS logs | | Fast `REFUSED` under load | `REFUSED` | 0 | 3, `forward` cap | forward reject counter | | Failures right after a restart, then normal | `NXDOMAIN` or timeout | climbing | 3, pod health | pod status and memory limit | Three of those rows describe a cluster where CoreDNS is healthy and clients are still broken. The `RESTARTS` value comes from pod status, not from the metrics endpoint, and the `first check` column names the artifact to run for each row, including the PromQL queries in the verification section. For hop 2, run the [`conntrack -S`](https://manpages.debian.org/bookworm/conntrack/conntrack.8.en.html) check on the node hosting the pod: [`insert_failed`](https://manpages.debian.org/bookworm/iproute2/lnstat.8.en.html) counts state entries the kernel failed to insert, and one increment per stall means the packet died before CoreDNS received it. ```bash sudo conntrack -S | grep insert_failed ``` * * * _**Reality Check: _**`kubectl rollout restart deployment/coredns -n kube-system`**_ flushes the cache and clears the symptom for a few hours. It leaves _**`ndots`**_, conntrack, and upstream capacity as they were, so the ticket returns.**_ * * * ## ndots:5 Turns One Lookup Into Eight Check hop 1 first. It is the cheapest to change and the easiest to misread. ```bash kubectl exec -it dnsutils -- cat /etc/resolv.conf ``` ```text nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` kubelet writes that file for every pod in `ClusterFirst` mode, and the [Kubernetes DNS documentation](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) shows the same three line default. `ndots` sets how many dots a name needs before the resolver tries it as an absolute name first; [resolv.conf(5)](https://man7.org/linux/man-pages/man5/resolv.conf.5.html) calls it the threshold “which must appear in a name given to res\_query(3) before an initial absolute query will be made” and puts the default at 1. Kubernetes raises it to 5 so short Service names resolve inside the cluster. `api.stripe.com` has two dots, so the resolver walks the search list first: 1. `api.stripe.com.default.svc.cluster.local` returns `NXDOMAIN` 2. `api.stripe.com.svc.cluster.local` returns `NXDOMAIN` 3. `api.stripe.com.cluster.local` returns `NXDOMAIN` 4. `api.stripe.com` returns `NOERROR` Each name goes out as an A query and an AAAA query, so one `getaddrinfo()` call produces eight lookups, or ten when the node appends its own search domain. Three answers can never exist, and CoreDNS still answers all of them, caches the denials, and forwards the fourth upstream on a miss. ### Lowering ndots Without Breaking Service Discovery For a workload that mostly calls external APIs, lower the threshold in the pod spec. `dnsConfig` merges into the options the DNS policy generates, and a duplicate option name replaces the default, so the pod’s `ndots:2` wins over the kubelet’s `ndots:5`: ```yaml apiVersion: v1 kind: Pod metadata: name: api-client spec: containers: - name: app image: /api-client: dnsConfig: options: - name: ndots value: "2" ``` Confirm the merge inside the running pod, because a mutating webhook can put the old value back. ```bash kubectl exec -it api-client -- cat /etc/resolv.conf ``` That manifest can break a StatefulSet lookup. With `ndots:2`, any name carrying two or more dots is tried as an absolute name first and only then against the search list. `orders.prod` has one dot and still walks the search list. `orders.prod.svc` has two, so it goes out absolute, fails, and retries through the search list. The [getaddrinfo(3)](https://man7.org/linux/man-pages/man3/getaddrinfo.3.html) call in glibc pays one extra round trip. [musl](https://musl.libc.org/), the C library Alpine ships, [suppresses the search list](https://git.musl-libc.org/cgit/musl/tree/src/network/lookup_name.c) once a name meets `ndots`, with no fallback, so `pod-0.orders.prod.svc` can stop resolving. Test those names before rolling the change out. ## The Exactly-Five-Second Stall Is a Kernel Race glibc sets the five second ceiling through its `timeout` default of five seconds and `attempts` default of two, so a dropped UDP query costs five seconds of silence before the retry opens a new socket. glibc sends the A and AAAA queries in parallel over one UDP socket, so both share a source IP and port. Both packets reach the DNAT rule for the `kube-dns` ClusterIP within microseconds of each other and both need a conntrack entry keyed on the same five-tuple. One creates the entry; the other loses the race inside the kernel’s confirm step and is dropped without an ICMP error. Five seconds later the retry uses a new source port, the five-tuple differs, and the answer arrives. CoreDNS logs nothing, its request duration histogram stays flat near a millisecond, and the restart count stays at zero, because a server that never received the packet has nothing to report. The two panels below follow the same five-tuple: on the left the second packet dies before CoreDNS sees it, and on the right the retry from a new source port succeeds. ![Conntrack race](https://adamtheautomator.com/wp-content/uploads/publisher/10ad90048a31c15f92db568d5466be590b964450f9a9637a7b59121a96c42798.jpg) ### The Per-Workload Fix for glibc For glibc containers, `single-request-reopen` closes the socket between the two queries, forcing the second onto a new source port and removing the collision. It is a second entry in the same `options` list as `ndots`, not a second `dnsConfig` block: ```yaml spec: dnsConfig: options: - name: ndots value: "2" - name: single-request-reopen ``` An option with no value is valid in `dnsConfig`, and the resulting `resolv.conf` carries the bare name. `single-request` sends the queries sequentially over one socket, which also removes the race and costs a full round trip every lookup. * * * _**Warning: _**`single-request-reopen`**_ is a glibc extension. musl ignores it, so an Alpine image with this option set behaves exactly as it did before. Rebuilding on a glibc base or moving to _**[_**NodeLocal DNSCache**_](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/)**_ are the two paths that work for Alpine workloads.**_ * * * ## When CoreDNS Itself Is the Problem An overloaded CoreDNS looks different: clients get `SERVFAIL` quickly, or the pods restart, or the cache stops helping because every query is a miss. Start with pod state. ```bash kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide ``` A `RESTARTS` count that climbs inside the incident window moves the investigation to memory limits and cache sizing, and `kubectl -n kube-system describe pod -l k8s-app=kube-dns` names `OOMKilled` in the last state when memory is the cause. The status column usually reads `CrashLoopBackOff` by the time anyone looks. Logs answer the next question: whether the failure happened while talking to an upstream. ```bash kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100 ``` Lines containing `plugin/forward` and `i/o timeout` point at hop 4. A `SERVFAIL` with no forward error usually means the `kubernetes` plugin cannot reach the API server. Neither is fixed by adding replicas, because replicas multiply the same broken upstream conversation. The [DNS debugging guide](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) covers the pod-level checks. ### A Corefile Worth Running This block keeps the default plugin order and changes the settings that matter during an incident. ```text .:53 { errors { consolidate 5m ".* i/o timeout$" } health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 cache 30 { success 9984 300 denial 9984 30 prefetch 10 60s 10% serve_stale 1h verify } forward . /etc/resolv.conf { max_concurrent 1000 failfast_all_unhealthy_upstreams } loop reload loadbalance } ``` In that block, the `30` in `cache 30` caps every cached response at 30 seconds, `success 9984 300` raises that cap to 300 seconds for positive answers, and `ttl 30` in the `kubernetes` stanza sets the record TTL, so a positive entry still expires at 30 seconds. `errors` with `consolidate` collapses repeated messages into one line per interval, so a storm of upstream timeouts does not flush the log buffer. `health` with `lameduck 5s` answers `200 OK` during shutdown, giving kube-proxy time to stop sending queries. `prometheus :9153` binds metrics to every interface, which is what a pod-IP scrape needs; see the verification section for why the default binding is not enough. The rest of the block is a set of choices, not boilerplate: * `kubernetes { ttl 30 }` raises the record TTL from its default of five seconds, so a Service IP change stays cached six times longer. * `pods insecure` is what the shipped kube-dns Corefile sets, while the plugin’s own default is `disabled`. It serves `1-2-3-4.ns.pod.cluster.local` records without checking that the address belongs to a live pod; `pods verified` adds that check by maintaining a watch on every pod and matching the requested address in memory, so it costs substantially more memory than `insecure` mode instead of an API call per lookup. * `prefetch 10 60s 10%` refreshes a popular entry before it expires once it has been queried 10 times with no gap longer than 60 seconds and its TTL drops below 10 percent. * `failfast_all_unhealthy_upstreams` returns `SERVFAIL` immediately once every upstream fails its health check, rather than sending the query to a random dead server. * `loop` halts CoreDNS at startup if the Corefile forwards to itself, `reload` picks up edits without a restart, and `loadbalance` randomizes the order of A, AAAA, and MX records in multi-answer responses. ### Size Both Cache Tables The cache holds two tables, one for successes and one for negative caching of denials, each split into 256 shards, and the [cache plugin documentation](https://coredns.io/plugins/cache/) states that “the default cache size is 9984 per cache”. The success table evicts randomly when a shard fills; the denial table evicts LRU. A cluster where every external name costs eight lookups fills the denial table first. | Directive | What it changes | Default value | | --- | --- | --- | | `success CAPACITY TTL` | Positive cache size and maximum TTL | 9984 entries, 3600s cap | | `denial CAPACITY TTL` | `NXDOMAIN` and `NODATA` cache size and TTL | 1800s cap, 5s minimum | | `serve_stale DURATION MODE` | Serving expired entries when the upstream fails | `immediate` refresh, one hour window | | `servfail DURATION` | How long a `SERVFAIL` stays cached | 5s, never more than five minutes | The `servfail` default explains a pattern that confuses on-call engineers: a lookup fails, then works, then fails again inside the same minute, because CoreDNS serves a cached `SERVFAIL` for five seconds. `serve_stale` covers the opposite case: `verify` asks the upstream first and falls back to the expired entry only on failure, while `immediate` answers from cache and refreshes in the background. ### Size CoreDNS From Measured Load Read the request rate before changing `replicas`: ```text sum(rate(coredns_dns_requests_total[5m])) ``` ```bash kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.replicas}' ``` Kubernetes scales CoreDNS from cluster size rather than query rate: the [cluster-proportional-autoscaler](https://kubernetes.io/docs/tasks/administer-cluster/dns-horizontal-autoscaling/) reads the `kube-dns-autoscaler` ConfigMap, whose documented defaults are `coresPerReplica: 256` and `nodesPerReplica: 16`. ```text replicas = max( ceil(cores / coresPerReplica), ceil(nodes / nodesPerReplica) ) ``` No published figure converts requests per second into replicas, so your measured rate is the budget. Override `nodesPerReplica` until the formula produces the count your rate implies. The memory limit covers both cache tables you sized above, 19,968 entries across `success 9984` and `denial 9984`, plus 1000 in-flight queries from `max_concurrent`. Set it above your peak `container_memory_working_set_bytes`. ### Forward: Order, Policy, and the REFUSED You Did Not Expect Multiple `forward` blocks are evaluated in the order they appear, so a stub zone has to come before the catch-all: ```text . { forward internal.example.com 10.100.0.2 10.100.0.10 { policy sequential failover SERVFAIL REFUSED } forward . /etc/resolv.conf { max_concurrent 1000 } } ``` The [forward plugin documentation](https://coredns.io/plugins/forward/) sets `policy` to `random` by default and `read_timeout` to 2s. Random selection sends roughly half your queries to the slow server first, and each of those waits out the full two seconds before the same query is retried against the healthy one. Half your lookups therefore gain two seconds, and upstream query volume rises by about 50 percent. A timeout already fails over on its own, and `failover` only extends that behavior to response codes, including `SERVFAIL` and `REFUSED`, which is not the problem a slow server creates. `policy sequential` holds traffic on the first upstream until health checks mark it down, which fits a primary and secondary pair. A burst of `REFUSED` at peak load means `max_concurrent` is too low, and `failfast_all_unhealthy_upstreams` turns the all-down case into an immediate `SERVFAIL` instead of a send to a dead server. ### autopath Is Not a Free Win `autopath` looks like the direct answer to `ndots:5`: CoreDNS resolves the search path server-side and returns a CNAME, collapsing several round trips into one. It is absent from the default `Corefile`, and the [plugin documentation](https://coredns.io/plugins/autopath/) opens with a warning that “There are several known issues, see the Bugs section below.” Those include resolving a recycled pod IP to the previous pod’s namespace, no Windows node support, and a fallthrough where a negative server-side answer sends the client back through every path manually. Skip it unless every client is Linux and you control the Corefile for the whole cluster: the `ndots` change above costs one option and leaves no server-side state to get wrong. ## NodeLocal DNSCache Removes the Race Entirely NodeLocal DNSCache changes the path every earlier fix merely patches: a DaemonSet runs CoreDNS in cache mode on each node, pods query it at a link-local address, and the query never crosses the DNAT rules. The Kubernetes page on [skipping iptables DNAT](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/) gives the reason: “Skipping iptables DNAT and connection tracking will help reduce conntrack races and avoid UDP DNS entries filling up conntrack table.” Deployment follows that manifest, with five substitutions: 1. Download the sample `nodelocaldns.yaml`. 2. Read the ClusterIP with `kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}'`. 3. Pick a local address, usually `169.254.20.10`. 4. Run the `sed` line for your kube-proxy mode. 5. Apply it and confirm one pod per node. For iptables mode, the documented substitution is: ```bash localdns=169.254.20.10 domain=cluster.local coredns=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}') sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$coredns/g" nodelocaldns.yaml kubectl create -f nodelocaldns.yaml ``` On macOS, BSD `sed` needs its own form: ```bash # BSD sed reads the argument after -i as a backup extension, so the GNU line fails with "undefined label" sed -i '' "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$coredns/g" nodelocaldns.yaml ``` In iptables mode the pod listens on both the ClusterIP and `169.254.20.10`, so running pods pick up the cache without a restart. In IPVS mode it can only bind the link-local address, so `--cluster-dns` has to change before pods use it. * * * _**Warning: a node-local-dns container that gets _**`OOMKilled`**_ does not clean up the packet-filter rules it installed at startup. Queries on that node keep pointing at a local address with nothing listening until the DaemonSet restarts the container, which means a short DNS outage on every crash. The default cache holds about 10,000 entries and uses roughly 30 MB when full, so size the limit from measured peak usage.**_ * * * ## Verifying the Fix with PromQL CoreDNS exposes metrics on port 9153 through the [CoreDNS metrics plugin](https://coredns.io/plugins/metrics/), and the endpoint binds to `localhost:9153` unless the `Corefile` names an address. [Monitoring a Kubernetes cluster with Prometheus and Grafana](https://adamtheautomator.com/prometheus-kubernetes/) covers the scrape side of that path. `kubectl port-forward` reaches that loopback binding, but a ServiceMonitor or any Prometheus pod scraping the CoreDNS pod IP gets connection refused until you add the `prometheus :9153` line shown above, which binds every interface. ```text sum by (rcode) (rate(coredns_dns_responses_total[5m])) ``` The split by `rcode` separates a `SERVFAIL` problem from an `NXDOMAIN` problem, which is the difference between a broken upstream and a broken search path. ```text histogram_quantile(0.99, sum by (le) (rate(coredns_dns_request_duration_seconds_bucket[5m]))) ``` A p99 above a few milliseconds means CoreDNS is queueing requests, which points at CPU limits, replica count, or a stalled upstream. The [Prometheus histogram guidance](https://prometheus.io/docs/practices/histograms/) covers how bucket boundaries cap what this query can resolve. ```text sum(rate(coredns_cache_hits_total{type="success"}[5m])) / sum(rate(coredns_cache_requests_total[5m])) ``` A hit ratio that drops after a change usually means TTLs shrank or new names started arriving. ```text sum(rate(coredns_forward_max_concurrent_rejects_total[5m])) ``` Any sustained value above zero means clients are receiving `REFUSED` because the forward queue is full. ### The Check That Covers Hop 2 The p99 above measures hop 3 only. A conntrack drop never reaches the process, so the histogram stays clean while clients stall, which is how a healthy CoreDNS dashboard coexists with a broken cluster. Close that gap with a timing loop in the pod that stalled. `nslookup` talks to its own DNS client, so it never reproduces the parallel socket or the glibc timeout that produce the stall. `getent hosts` uses `gethostbyname2`, so it issues A and AAAA lookups in sequence. `getent ahosts` is the `getaddrinfo()` variant and the one that reproduces glibc’s parallel A/AAAA socket: ```bash kubectl exec -it api-client -- sh -c 'for i in $(seq 1 20); do time getent ahosts api.stripe.com > /dev/null; done' ``` Count the runs whose `real` value exceeds one second. A clean cluster returns zero, and the loop is cheap enough to rerun after each change. The image supplies both `getent` and the `time` keyword. If `sh -c` in that image is dash, `time` exits with `not found` before the loop measures anything, so run the same loop under `bash -c` or time the outer `kubectl exec` with `date +%s` before and after. ## Frequently Asked Questions Each question is answered above. ### Why Does a DNS Lookup Take Exactly 5 Seconds? glibc waits five seconds before retrying a dropped query. The retry’s new source port gets a fresh conntrack entry and an answer. ### Does Lowering ndots Break StatefulSet Lookups? At `ndots:2`, `pod-0.orders.prod.svc` goes out absolute first, and musl takes that as final, so it stops resolving. ### Do I Need NodeLocal DNSCache in Every Cluster? No. Deploy it when workloads run on musl, where `single-request-reopen` does nothing. It also pays off when DNS entries outpace `nf_conntrack_max`. ### What Does SERVFAIL Mean Here? A fast `SERVFAIL` with no forward error comes from the [kubernetes plugin](https://coredns.io/plugins/kubernetes/), which answers `SERVFAIL` for Kubernetes records it has not synchronized; check the plugin’s API server connection before you touch the upstream. `plugin/forward` plus `i/o timeout` lines point at the upstream. ## What to Change First Sort the work by cost and reversibility, and stop when the measurements say the incident is over. 1. Classify the failure with the decision table: 5.00s stall, scattered latency, fast `SERVFAIL`, or `REFUSED`. 2. Lower `ndots` to 2 on workloads that mostly call external names, after testing StatefulSet and headless Service lookups. 3. Add `single-request-reopen` to glibc workloads that show the 5.00s stall, and skip it for Alpine. 4. Tune the `Corefile`: denial cache size, `serve_stale`, forward ordering, and `max_concurrent`. 5. Size CoreDNS replicas from the measured query rate and set the memory limit to fit the cache. 6. Deploy NodeLocal DNSCache when both the race and the query volume point at hop 2. 7. Keep the client-side timing loop as the acceptance test for every change on this list. The five second number separates a kernel drop from a capacity problem, and the request duration histogram proves which one you are looking at. Find the shape first, trace it to the hop that produced it, and the next incident starts with a measurement instead of a restart. Read the [cache](https://coredns.io/plugins/cache/) and [forward](https://coredns.io/plugins/forward/) references end to end before you edit a production `Corefile`. Share this article [Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Ffix-coredns-timeouts-kubernetes%2F&text=Resolve%20Kubernetes%20CoreDNS%20Timeouts%20Like%20a%20Pro)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Ffix-coredns-timeouts-kubernetes%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Ffix-coredns-timeouts-kubernetes%2F) ## Related Posts ![](https://adamtheautomator.com/wp-content/uploads/publisher/3ec5d9c85b2b81d18bcdd4e82db4a47e/3655f2c9e9bdd72484d7c264e373fd1aaa68e5b72b9454cd575b1e0eb08f2063.png) ### [Kubernetes Pending Pods: Fix Scheduling and Resource Limits](/kubernetes-pending-pods-scheduling/) Decode why Kubernetes pods stay Pending: read the FailedScheduling event, then fix requests, taints, node affinity, storage, or namespace quota. ![](https://adamtheautomator.com/wp-content/uploads/2026/07/featured_image-3.png) ### [Build Your First Internal Developer Platform](/build-first-internal-developer-platform/) Build your first Internal Developer Platform with Backstage, a software catalog, software templates, CI/CD handoffs, and Kubernetes deployment manifests. ![](https://adamtheautomator.com/wp-content/uploads/2026/06/featured_image-9.png) ### [DevOps to Platform Engineer: 2026 Transition Roadmap](/devops-platform-engineer-2026-transition-roadmap/) Learn how to move from DevOps to platform engineering in 2026 with a practical roadmap covering transferable skills, internal developer platforms, Backstage, Crossplane, and portfolio projects. ## Categories * [IT Ops](/category/it-ops/) * [Cloud](/category/cloud/) * [DevOps](/category/devops/) * [Home Ops](/category/home-ops/) * [Information Security](/category/infosec/) * [Software Development](/category/software-development/) ## Site * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)