--- title: "Kubernetes Pending Pods: Fix Scheduling and Resource Limits" description: "Decode why Kubernetes pods stay Pending: read the FailedScheduling event, then fix requests, taints, node affinity, storage, or namespace quota." canonical: "https://adamtheautomator.com/kubernetes-pending-pods-scheduling/" --- # Kubernetes Pending Pods: Fix Scheduling and Resource Limits > Decode why Kubernetes pods stay Pending: read the FailedScheduling event, then fix requests, taints, node affinity, storage, or namespace quota. Source: https://adamtheautomator.com/kubernetes-pending-pods-scheduling/ --- ATA Learning Tap to hide [ ATA Learning ](/) * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Search for: * [](https://twitter.com/adbertram) * [](https://github.com/Adam-the-Automator) * [](https://www.linkedin.com/company/adam-the-automator-llc) * [](/feed/) ![Kubernetes Pending Pods: Fix Scheduling and Resource Limits](https://adamtheautomator.com/wp-content/uploads/publisher/3ec5d9c85b2b81d18bcdd4e82db4a47e/3655f2c9e9bdd72484d7c264e373fd1aaa68e5b72b9454cd575b1e0eb08f2063.png) # Kubernetes Pending Pods: Fix Scheduling and Resource Limits [![](https://secure.gravatar.com/avatar/d0b9d42e21e5622713f8b693aa5c0f9244d5f7dd200ed29b8398f52dee5de337?s=192&d=mm&r=g)Adam Bertram](https://adamtheautomator.com/author/adam-bertram/)7 October 20269 min. read Categories: [DevOps](/category/devops/) Tags:[Kubernetes](/tag/kubernetes/)[Containers](/tag/containers/)[DevOps](/tag/devops/) Table of Contents * [What Pending Actually Means Before You Touch Anything](#what-pending-actually-means-before-you-touch-anything) * [How the Scheduler Filters Nodes](#how-the-scheduler-filters-nodes) * [Why a Memory Limit Can Still Sink a Pod](#why-a-memory-limit-can-still-sink-a-pod) * [Step 1: Read the FailedScheduling Event Like a Scheduler Log](#step-1-read-the-failedscheduling-event-like-a-scheduler-log) * [Decode the Node Tally](#decode-the-node-tally) * [Cause 1: Insufficient CPU or Memory Against Node Allocatable](#cause-1-insufficient-cpu-or-memory-against-node-allocatable) * [Right-Size the Request From Observed Usage](#right-size-the-request-from-observed-usage) * [Cause 2: Taints the Pod Does Not Tolerate](#cause-2-taints-the-pod-does-not-tolerate) * [Cause 3: nodeSelector and Affinity That Match No Node](#cause-3-nodeselector-and-affinity-that-match-no-node) * [Hard Rules Versus Soft Rules](#hard-rules-versus-soft-rules) * [Cause 4: An Unbound PersistentVolumeClaim and Zone Conflicts](#cause-4-an-unbound-persistentvolumeclaim-and-zone-conflicts) * [Cause 5: ResourceQuota and LimitRange Blocks](#cause-5-resourcequota-and-limitrange-blocks) * [The Triage Order That Stops the Guesswork](#the-triage-order-that-stops-the-guesswork) * [Read the Pod Status First](#read-the-pod-status-first) * [Decode the Event, Then Pick the Cause](#decode-the-event-then-pick-the-cause) * [Stop the Next Pending Pod Before It Starts](#stop-the-next-pending-pod-before-it-starts) * [Wire the Guardrails Into the Manifest](#wire-the-guardrails-into-the-manifest) Three nodes. Forty percent free memory on every dashboard you open. And a Deployment that sits in `Pending` while `kubectl describe` reports `0/3 nodes are available`. Before you provision a fourth node or double a memory limit, read the message the scheduler already recorded. A Pending pod is the scheduler’s answer to an arithmetic question about requests, and that answer names the exact term your manifest got wrong. This post gives you a fixed order for reading that answer: start from the pod’s phase, read the `FailedScheduling` event it produced, then follow that event to the one resource it names. Get the diagnosis right and each fix is one edit. ## What Pending Actually Means Before You Touch Anything Tested on Kubernetes v1.31 with kubectl v1.31. Kubernetes tracks each pod through a small set of phases, and [Pending](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) is the first. The API server has accepted the pod, but the scheduler has not bound it to a node. That window covers the scheduler’s evaluation and the node’s image pull. Once a node is chosen and the images are present, the pod moves to `ContainerCreating` and then `Running`. The split tells you which system to blame. Run `kubectl get pod` and read the `NODE` and `STATUS` columns together. * `Pending` with an empty `NODE` column: the scheduler rejected every node. Your problem is a filter-pass constraint. * `Pending` with a node name, or `ContainerCreating`: the scheduler succeeded. Your problem is an image pull, a volume mount, or a CNI plugin. * `ImagePullBackOff` or `ErrImagePull`: the node accepted the pod and the registry refused the image. ### How the Scheduler Filters Nodes Each scheduling cycle runs two passes. A filter pass strips out every node that fails a hard constraint: not enough allocatable CPU or memory, a taint the pod does not tolerate, a node label it requires, or a volume it cannot attach. A score pass ranks the survivors and binds the pod to the highest-scoring node that the [kube-scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) selected. A pod stuck in Pending means the filter pass returned zero nodes. Requests are the numbers that survive the filter. A request is the slice of CPU or memory the scheduler reserves on a node for that pod. Kubernetes’ [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) page states the mechanism plainly: “When you specify the resource request for containers in a Pod, the kube-scheduler uses this information to decide which node to place the Pod on.” Live usage from your metrics never appears in that sentence. ### Why a Memory Limit Can Still Sink a Pod Limits are enforced after startup by the kubelet through CPU throttling and memory OOM kills, separate from the scheduler’s placement decision. A limit on its own does not block placement. A limit with a blank request does, because Kubernetes copies the limit into the request when you leave the request empty. A container with `limits.memory: 4Gi` and no `requests.memory` reaches the scheduler asking for 4Gi. A node with 1Gi free answers `Insufficient memory`, and the blame lands on a number you thought was only a ceiling. * * * _**Warning: The implicit request copy is the most common self-inflicted Pending pod. Declare a request on every container, even when you also declare a limit, so the scheduler sees the number you intended.**_ * * * ![Scheduler filter and score](https://adamtheautomator.com/wp-content/uploads/publisher/7a57b94ae4b803ec06b02cd167bf5a12ff82e4f3b66454ec4f4a2afd33cc255e.png) ## Step 1: Read the FailedScheduling Event Like a Scheduler Log Start by listing the pods that need attention. [Field selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/) let you filter server-side instead of listing every pod in the namespace: ```bash kubectl get pods -n --field-selector=status.phase=Pending ``` Then describe the pod and read its event log, which carries the whole diagnosis: ```bash kubectl describe pod -n ``` Scroll to the `Events` block at the bottom. When the scheduler rejected every node, it writes a `FailedScheduling` warning. The text below is the canonical shape of that event; the node counts and reasons change with your cluster. ```text Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 42s default-scheduler 0/3 nodes are available: 2 Insufficient memory, 1 node(s) had untolerated taint {dedicated: gpu}. ``` ### Decode the Node Tally Read the message as a census of rejected nodes. The `0/3` means zero nodes passed the filter out of three considered. Everything after the colon is a per-reason tally of rejected nodes. In the example above, two nodes lacked allocatable memory and one carried a taint the pod could not tolerate. If all three had failed for the same reason, the message would read `3 Insufficient memory`. Read the tally as a priority list. Fix the reason covering the most nodes first, because one change often clears the pod. * * * _**Pro Tip: _**`kubectl describe`**_ reads from the namespace in your current context. Run _**`kubectl config view --minify`**_ if the output looks empty when you know the pod exists.**_ * * * ## Cause 1: Insufficient CPU or Memory Against Node Allocatable An `Insufficient cpu` or `Insufficient memory` tally means the node’s ledger is full, whatever the live hardware is doing. The scheduler compares your request against the node’s allocatable capacity: total resources minus what the kubelet and system daemons reserve. A node can show 5 percent real CPU usage and still reject a pod, because 100 percent of its CPU is already requested by idle replicas. Confirm the ledger before changing anything: ```bash kubectl describe node ``` Look for the `Allocated resources` block. It reports requested CPU and memory as a percentage of allocatable, listed separately from the limits. A node at or near 100 percent requested is full from the scheduler’s point of view, whatever the live metric says. The [reserve compute resources](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/) task explains how the kubelet carves allocatable out of capacity. ### Right-Size the Request From Observed Usage The fix is usually to correct the request, not to add a node. Pull real usage first. `kubectl top` reads from the [resource metrics pipeline](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/): ```bash kubectl top pod -n --sort-by=memory ``` Compare live usage against the requested value in the deployment. The table maps the common mismatches; the figures are illustrative shapes rather than targets for your workload. | Symptom | What the scheduler sees | First move | | --- | --- | --- | | Pod Pending, node shows low real usage | Requests sum near 100 percent of allocatable | Lower `requests.cpu` or `requests.memory` to match observed peaks | | Pod Pending only at peak hours | Real demand outgrew the fixed request | Raise the request, then let the autoscaler add a node | | Pod Pending with no request set | The limit was copied into the request | Set an explicit request below the limit | | GPU pod Pending | Extended resources cannot be overcommitted | Free a device or add a GPU node | ## Cause 2: Taints the Pod Does Not Tolerate A node taints itself to repel pods it was not built for. Control-plane nodes carry one, and so do spot instances and GPU pools. A pod without a matching toleration is filtered out, and the event names the taint. List every taint in the cluster: ```bash kubectl describe nodes | grep -i Taints ``` Then compare the taints against the pod’s `spec.tolerations`. The [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) documentation defines three effects, and only two of them can hold a pod in Pending: | Effect | What it does to new pods | Causes Pending? | | --- | --- | --- | | `NoSchedule` | Blocks any pod without a matching toleration | Yes | | `PreferNoSchedule` | Tries to avoid the node, but schedules if nothing else fits | Rarely | | `NoExecute` | Blocks new pods and evicts running pods that do not tolerate it | Yes | Match the tolerance to the taint exactly. A toleration needs the same key and effect, and `Equal` also needs the same value. When the taint was applied by mistake, remove it with a trailing minus sign instead of editing every deployment: ```bash kubectl taint nodes dedicated=gpu:NoSchedule- ``` ## Cause 3: nodeSelector and Affinity That Match No Node Taints push pods away from nodes; affinity pulls them toward nodes. A hard node-affinity rule or a `nodeSelector` is a filter that runs before scoring, and a label that exists on zero nodes strands the pod forever. The event is specific about it: `didn't match Pod's node affinity/selector`. Print the labels your cluster actually carries: ```bash kubectl get nodes --show-labels ``` Then read the constraint the pod asks for: ```bash kubectl get pod -n -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity}{"\n"}' ``` ### Hard Rules Versus Soft Rules The [assigning pods to nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) reference draws the line that decides whether a mismatch strands the pod. `requiredDuringSchedulingIgnoredDuringExecution` is a filter: the pod cannot schedule unless the rule is met. `preferredDuringSchedulingIgnoredDuringExecution` is a score: when no node matches, the scheduler places the pod anyway. A hard rule that matches no node is a permanent Pending pod. A soft rule is a nudge. When the workload can run outside its preferred zone, downgrade the hard rule to a preference. When it genuinely cannot, label a node to match before you deploy. ## Cause 4: An Unbound PersistentVolumeClaim and Zone Conflicts Stateful pods add a storage filter to the scheduling cycle. If the pod references a claim that is not bound to a volume, the scheduler cannot place it, and the event reads `pod has unbound immediate PersistentVolumeClaims` or `persistentvolumeclaim "data-pvc" not found`. Shift the investigation to the claim. List the claims in the namespace first: ```bash kubectl get pvc -n ``` Then describe the claim that will not bind: ```bash kubectl describe pvc -n ``` A claim stuck in `Pending` moves the problem to provisioning. Read the claim’s own events; the usual causes are a missing `storageClassName`, a provisioner that lacks cloud permissions, or a volume bound to a zone with no schedulable node. The [Kubernetes persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) reference and the [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) reference cover binding modes. One binding mode removes a whole class of trap. `volumeBindingMode: WaitForFirstConsumer` on the [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) tells the provisioner to wait until the scheduler picks a node, then create the volume in that node’s zone. With `Immediate` binding, the volume can land in a zone where no node has free capacity, and the pod pends against a topology mismatch no request change can fix. ## Cause 5: ResourceQuota and LimitRange Blocks Some Pending pods never reach the scheduler. A Deployment is accepted, but its ReplicaSet fails to create pods, and `kubectl get pods` shows nothing because the pods were never admitted. Check the ReplicaSet events for the real message: ```bash kubectl describe replicaset -n ``` A quota rejection reads `Forbidden: exceeded quota`. The [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) concept explains why: once a quota covers `cpu` or `memory`, every new pod must declare a request or limit for it, and the control plane rejects an over-quota pod with HTTP `403 Forbidden`. The admission controller rejected the pod before the scheduler ever saw it, which is why no `FailedScheduling` event appears. Look at the namespace ceiling and the injected defaults. Read the quota first: ```bash kubectl describe quota -n ``` Then read the LimitRange that supplies the defaults: ```bash kubectl describe limitrange -n ``` A [LimitRange](https://kubernetes.io/docs/concepts/policy/limit-range/) silently injects default requests and limits into any container that omits them. Those defaults count against the quota, so a namespace can exhaust its budget from pods whose authors never set a value. If `kubectl describe quota` shows `requests.cpu` near its hard limit while `kubectl top` shows almost no real usage, the gap is requests that no one sized on purpose. ## The Triage Order That Stops the Guesswork Run the diagnosis in one fixed sequence and you will not chase the wrong layer. Start with the pod status, read the event, then follow the event to its owner. ### Read the Pod Status First 1. List Pending pods with `kubectl get pods --field-selector=status.phase=Pending`. 2. Run `kubectl describe pod` and read the `Events` block. 3. If the event is `FailedScheduling`, decode the node tally and go to the matching cause below. 4. If there is no pod object at all, describe the owning ReplicaSet for a quota or admission rejection. 5. If the pod has a node and is still not running, leave scheduling behind and debug the image or volume. ### Decode the Event, Then Pick the Cause The table maps the event string you will most often see to its cause and first command. It covers the paths above plus two worth knowing: pods that pend because the node hits its per-node pod ceiling, and pods that cannot preempt their way to capacity because a [PodDisruptionBudget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) protects the running replicas. | Event or symptom | Root cause | First command | | | --- | --- | --- | --- | | `N Insufficient cpu` / `Insufficient memory` | Requests exceed node allocatable | `kubectl describe node` (Allocated resources) | | | `node(s) had untolerated taint` | Missing or mismatched toleration | \`kubectl describe nodes \\ | grep -i Taints\` | | `didn't match Pod's node affinity/selector` | Required label on no node | `kubectl get nodes --show-labels` | | | `unbound immediate PersistentVolumeClaims` | Claim not bound or zone-locked | `kubectl describe pvc` | | | `Too many pods` | Per-node pod ceiling reached | `kubectl describe node` (Non-terminated Pods) | | | `No preemption victims found for incoming pod` | Low priority and no evictable pods | Check PriorityClass and PodDisruptionBudgets | | | `Forbidden: exceeded quota` | Namespace quota or LimitRange default | `kubectl describe quota` | | ![Pending pod decision tree](https://adamtheautomator.com/wp-content/uploads/publisher/14a5ccb84455a9953ae5ddfe9728f49503fca43e4c59dbc042b0589d43bba21a.png) ## Stop the Next Pending Pod Before It Starts Every cause above is cheaper to prevent than to debug at 2 a.m. Set requests from observed usage, and set them explicitly so no limit is copied in behind your back. Give each namespace a `LimitRange` with sane defaults, so a container that omits a value inherits a small one instead of starving the quota. Downgrade node-affinity rules to preferences where the workload can move, and keep one required rule only when placement is a real constraint. ### Wire the Guardrails Into the Manifest For capacity that grows with demand, pair a Horizontal Pod Autoscaler with a provisioner such as the [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) or [Karpenter](https://karpenter.sh/), and confirm the provisioner’s maximum size is not the reason a pod sits Pending after the request was fixed. [Pod priority and preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/) raise a pod above the queue, but a `PodDisruptionBudget` can block the eviction preemption needs, so a high-priority pod can still pend with `No preemption victims found`. Finally, alert on the phase itself. A Prometheus rule on `kube_pod_status_phase{phase="Pending"}` that fires after five minutes catches the pods that never generate a page from a failing service, and the [node-pressure eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/) rules tell you which pods a stressed node will shed first. The scheduler records the reason in the event log when it gives up, and the fix is almost always a request, a toleration, a label, a binding mode, or a quota line you can point to in one command. Share this article [Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fkubernetes-pending-pods-scheduling%2F&text=Kubernetes%20Pending%20Pods%3A%20Fix%20Scheduling%20and%20Resource%20Limits)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fkubernetes-pending-pods-scheduling%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fkubernetes-pending-pods-scheduling%2F) ## Related Posts ![](https://adamtheautomator.com/wp-content/uploads/2026/07/featured_image-3.png) ### [Build Your First Internal Developer Platform](/build-first-internal-developer-platform/) Build your first Internal Developer Platform with Backstage, a software catalog, software templates, CI/CD handoffs, and Kubernetes deployment manifests. ![](https://adamtheautomator.com/wp-content/uploads/2026/06/featured_image-9.png) ### [DevOps to Platform Engineer: 2026 Transition Roadmap](/devops-platform-engineer-2026-transition-roadmap/) Learn how to move from DevOps to platform engineering in 2026 with a practical roadmap covering transferable skills, internal developer platforms, Backstage, Crossplane, and portfolio projects. ![](https://adamtheautomator.com/wp-content/uploads/publisher/2e05d9c85b2b8187b503fb0f6ceff47e/e15d54857176da3fe72b1f0692faaa7000367a5fbe6edf5f6a5f1034395fea45.webp) ### [Azure Bicep: Bulletproof Production Deployment Patterns](/azure-bicep-production-patterns/) Own your Bicep blast radius: pin modules in a private registry, gate pull requests with lint and PSRule, and track production changes with stacks. ## Categories * [IT Ops](/category/it-ops/) * [Cloud](/category/cloud/) * [DevOps](/category/devops/) * [Home Ops](/category/home-ops/) * [Information Security](/category/infosec/) * [Software Development](/category/software-development/) ## Site * [Home](/) * [Tutorials](/tutorials/) * [Instructors](/author/) * [Advertising](/advertising/) * [Recommended Resources](/resources/) * [About Adam](/about-adam/) Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)