Operations

Kubernetes Pods vs Nodes: What Runs Where—and Why Scheduling Fails

A Pod is Kubernetes’ deployable workload unit. A Node is the worker machine that supplies the environment and resources to run it. When a Pod won’t run, check which Node it was assigned to and read its events. The cause is usually one of three things: resource requests, a Node eligibility rule, or a startup dependency such as storage. Fix the layer the evidence points to.

Kubernetes Pod vs Node: the difference at a glance

Kubernetes pod node

A Pod is a unit of work that Kubernetes deploys. A Node is the physical or virtual machine that runs it. One Node can host many Pods, and each Pod holds one or more containers.

DimensionPodNode
RoleDeployable workload unitWorker compute machine
ContainsOne or more containers, shared networking and optional volumesHost resources, kubelet and container runtime
Created or managed byA user or workload controllerCluster infrastructure management
SchedulingAssigned to an eligible NodeHosts multiple assigned Pods
Health informationPhase, conditions and container statesReady/NotReady, pressure conditions and resources
Typical problemsScheduling failure, container startup or volume errorsResource pressure, connectivity or host failure
First object to inspectPod status and eventsNode conditions and allocatable resources

Take a cluster running a web API and a logging agent. One Node hosts an API Pod and an agent Pod. The API Pod has an application container and a helper container that share the Pod’s network namespace. Containers in one Pod share a workload context. Separate Pods on one Node compete for its resources.

Pod container layout

An unscheduled Pod has no Node yet. Once the scheduler binds it, the assignment names the machine responsible for running its containers.

Two read-only commands show both objects:

kubectl get nodes -o wide
kubectl get pods -A -o wide

The first lists Nodes. The second lists Pods across accessible namespaces, with a NODE column showing the assigned machine when one exists.

If containers are new to you, start with how Docker containers differ from their host environment (for Docker images, container networks, and persistent volumes). Kubernetes adds workload scheduling and lifecycle management around those containers.

How Kubernetes schedules a Pod onto a Node

Kubernetes scheduling flow

The scheduler picks a Node for an unscheduled Pod. The kubelet on that Node then works with the container runtime to start and maintain the Pod’s containers.

In a typical Deployment, four actors are involved:

  1. The workload controller creates the Pod. A Deployment manages ReplicaSets, which keep the intended number of Pods running. You can also create a standalone Pod directly.
  2. The scheduler evaluates Nodes. It weighs resource requests and placement rules to decide where the Pod can run.
  3. The Pod gets a Node. Kubernetes records the choice in the Pod’s spec.nodeName field.
  4. The kubelet runs it. The assigned Node prepares the containers and their dependencies, then reports status through Kubernetes.

Say a Deployment has three replicas. Two Pods get Nodes and the third stays unscheduled. The controller did its part, but the scheduler still has no destination for that third Pod.

Scheduler node assignment

A Pod stays unassigned when every candidate Node fails a scheduling requirement.

To check the assignment for one Pod:

kubectl get pod POD -n NAMESPACE \
  -o jsonpath='{.spec.nodeName}'

Replace POD and NAMESPACE with the actual names. A returned Node name is the assigned machine. An empty value means the Pod has no spec.nodeName yet.

Why free CPU doesn’t mean a Pod will fit

CPU requests capacity

A Node can look mostly idle and still lack schedulable CPU for another Pod. Kubernetes tracks three separate numbers: the machine’s capacity, its allocatable resources and the requests already assigned to workloads.

Monitoring tools show CPU usage over a measurement interval. The scheduler compares the Pod’s declared requests against the request budget that remains, along with other eligibility rules.

Take a Node with:

  • Allocatable CPU: 4 cores available for scheduling.
  • Existing Pod requests: 3.5 cores already requested by assigned workloads.
  • New Pod request: 1 additional core.

That leaves 0.5 cores of request headroom. Scheduling the new Pod would need 4.5 cores of requests against an allocatable budget of 4.

Requested CPU arithmetic

Requests drive scheduling and resource allocation. Limits cap runtime resource use, and how they’re enforced depends on the resource. A CPU limit does a different job from the CPU request used during placement. Memory follows the same split: requests and usage diverge, and resource pressure can affect the application while it runs.

The Kubernetes resource management documentation explains how requests, limits and allocation relate.

To see the scheduler’s resource picture for a Node:

kubectl describe node NODE

Check Allocatable and Allocated resources, especially the requested CPU and memory totals, and compare them with the new Pod’s requirements. kubectl top shows live usage when a metrics source is running. describe node shows what the scheduler counts.

GPU requirements, Pod-count limits and other scheduler filters can also exclude a Node on their own.

Node affinity, taints and volumes: three ways placement can fail

Kubernetes placement constraints

Spare resources make a Node a candidate. Placement rules decide whether the Pod may use it.

  • Node selectors and affinity: Required Node labels must match the Pod’s placement requirements. Preferred affinity steers the choice among eligible Nodes without necessarily excluding the rest.
  • Taints and tolerations: A Node’s taints repel Pods. A matching toleration lets the Pod be considered, and resource and placement rules still apply.
  • Persistent-volume topology: A volume can have locality or attachment limits that keep its Pod off an otherwise suitable Node.

Say a Pod requires the Node label disk=ssd. A Node without that label is out, even with spare CPU and memory.

Storage adds another restriction. A Pod can require a persistent volume whose placement rules clash with the Node picked by the workload’s affinity configuration.

Affinity scheduling conflicts

One Longhorn user ran into this with dedicated storage Nodes, worker Nodes and an LLM Node. Adding affinity rules produced an event that listed several separate exclusion categories.

If I create a test pod without any affinity rules, then the pod picks a node (e.g. a worker) and happily creates a PVC and longhorn manages this correctly. The moment I add an affinity rules (e.g. run ollama on the LLM node, create a pod that needs a PVC on the worker nodes only), the pod gets stuck in “pending” state and refuses to start because of: “0/8 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had volume node affinity conflict, 4 node(s) didn’t match Pod’s node affinity/selector. preemption: 0/8 nodes are available: 8 Preemption is not helpful for scheduling.”

r/kubernetes

The author later traced the problem to indentation under the YAML values entry. After fixing it, the example worked.

Pod affinity YAML

To see the Pod’s effective placement settings, inspect its YAML:

kubectl get pod POD -n NAMESPACE -o yaml

Compare its nodeSelector, required affinity, tolerations and referenced persistent-volume claims with the target Node’s labels, taints and volume topology. Redact cluster-specific values before sharing the output.

The Kubernetes taints and tolerations documentation describes how these controls affect eligibility.

A Pending Pod: a safe, fast diagnostic route

Pending pod diagnostics

Pending covers two situations. An unassigned Pod may be waiting for the scheduler. An assigned Pod may be waiting for images or storage while its containers are prepared.

Read the Pod’s events:

kubectl describe pod POD -n NAMESPACE

The output shows the assigned Node, relevant conditions, container information and an Events section with the reasons for the Pod’s current state.

ObservationInspect firstLook next at
No assigned Node; FailedScheduling eventResource requests and scheduler exclusionsEligible Nodes and placement constraints
Node assigned; image or container waiting eventsContainer startup and image messagesKubelet and runtime dependencies
Node assigned; volume mount or attachment eventsVolume-related messagesStorage availability and topology
FailedScheduling event reasons

In a February 2025 Discuss Kubernetes post, an operator reported 0/176 nodes are available for a Pod that would not schedule, despite an apparently unused machine.

The event listed 14 insufficient-CPU, 14 insufficient-memory and 14 GPU-related exclusions, plus 34 Node affinity or selector mismatches and six volume-affinity conflicts. The operator wanted to know why one particular Node was excluded. The thread never found a root cause.

Who controls placement when you can’t edit the Pod?

Pod controller ownership

A controller-generated Pod inherits its configuration from the workload that manages it, and admission policies can also change the effective Pod spec. With read access you can spot an unsuitable placement rule even if you can’t change the controlling template or cluster policy. Find the direct owner by kind and name:

kubectl get pod POD -n NAMESPACE -o jsonpath='{range .metadata.ownerReferences[*]}{.kind}/{.name}{"\n"}{end}'

Then identify the matching workload or policy owner. For a Deployment-managed Pod, the direct owner is typically a ReplicaSet. Follow that ReplicaSet’s owner reference to the Deployment before editing its Pod template.

What happens to Pods when a Node fails?

Kubernetes node failure

A Pod’s Node assignment is part of its lifecycle. When that machine fails, a controller typically creates a replacement Pod that can be scheduled on an eligible Node. The checks for uptime alerts, incident response, and restore tests are explained in the website reliability diagnosis and recovery playbook.

Restarting a container inside a Pod, replacing a Pod object and replacing the underlying Node are three different operations. They affect different Kubernetes objects and treat application state differently.

A Deployment running a web service creates a replacement when a replica disappears. The scheduler then has to find a Node with the required resources and dependencies.

Pod replacement lifecycle

A standalone Pod has no controller-managed replicas, and local or persistent storage requirements can limit what a replacement can access. Even a replacement that receives a Node can stay unavailable if a required volume can’t be attached or mounted.

In April 2022, an operator running Kubernetes 1.21.6 on GCP reported NFS mount failures on Discuss Kubernetes after a Node-pool replacement. The workaround: cordon the Nodes with mount errors and delete the affected Pods so workloads were scheduled on other Nodes.

For controller-managed workloads, check desired and available replicas:

kubectl get deployment DEPLOYMENT -n NAMESPACE

Compare the two counts. A gap means recovery is incomplete, and the associated Pod states show where.

Scale Pods or Nodes? Find what’s actually short

Kubernetes scaling decisions

Pod scaling and Node scaling fix different shortages. Which one you need depends on whether the application needs more replicas, the cluster needs more capacity, or a placement rule blocks scheduling.

A Horizontal Pod Autoscaler adjusts replica demand based on its configuration and available metrics. Node autoscaling changes machine capacity when a compatible provisioner and its policies allow it. Adding replicas creates more Pending Pods when the cluster has no room. Adding machines fixes a real capacity shortage when the new Nodes meet the workload’s requirements.

SymptomWhat to tryCheck
Too few application replicas; eligible Node capacity existsWorkload or HPA scalingReplica demand and scaling configuration
Insufficient request headroom; suitable Nodes can be provisionedNode scalingAvailable Node types, policies and workload compatibility
Required labels, taints or volume topology cannot be satisfiedAuthorized configuration correctionSpecific failed placement requirement
Very rapid creation of short-lived JobsJob lifecycle and object-churn investigationCleanup, workload design and infrastructure limits
Short job pod churn

Memory isn’t the only limit. In one Reddit post, an operator dedicated a Node with approximately 1 TB of RAM to a particular workload. Their controller launched short Jobs, but active, completed and Pending Pods piled up.

We have a controller pod which itself can trigger new jobs. Some jobs run for 1-3 seconds, but the kubelet is not keeping up with cleaning the pods and a lot of new jobs (100+) are left on `pending` as it is already running with 30 active jobs and 220 completed (but not deleted). How can I configure the kubelet to delete pods as soon as they are completed/finished? If I raise the limit of pods to 1000 we will have even more jobs on `pending` state?

r/kubernetes

1,000 Pods was the operator’s target, not a Kubernetes limit. Pod count, CNI allocation, Job objects and cleanup operations can each become the bottleneck.

Pod scaling capacity

Before changing scale, check whether the workload already has an HPA:

kubectl get hpa -A

Then find the Node autoscaler or provisioning mechanism your platform uses.

Leave a Reply

Your email address will not be published. Required fields are marked *