Kubernetes Disk Pressure: Why Nodes Fill Up and Prevention A platform engineer gets paged for evicted pods. They SSH into the node, run du on /var/log, and see plenty of room. Yet the node says DiskPressure=True.

That's because DiskPressure is a kubelet protection signal. It tracks available bytes and inodes on the filesystems kubelet monitors, and it isn't the same thing as a full persistent volume. This article covers what fills nodes, what happens when pressure persists, and the controls that keep it from happening.

Key Takeaways

  • DiskPressure means a monitored nodefs or imagefs crossed a byte or inode eviction threshold.
  • Logs, images, ephemeral storage, inode exhaustion, and mount layout cause most cases.
  • Kubelet reclaims first, then evicts pods and blocks new scheduling.
  • Prevention combines log rotation, ephemeral-storage limits, image GC, and trend alerts.

Common Causes of Kubernetes Disk Pressure

Kubelet evaluates available bytes and inodes on the filesystems used for node data, images, logs, and ephemeral storage. According to the Kubernetes node-pressure documentation, upstream hard defaults are:

Signal Hard threshold
nodefs.available <10%
imagefs.available <15%
nodefs.inodesFree <5% (Linux)
imagefs.inodesFree <5% (Linux)

These are upstream defaults. Managed services may configure them differently. The symptom is low storage, but the source could be a workload, the runtime, a log collector, or the filesystem layout.

Application and container logs

Kubelet rotates runtime-written stdout and stderr logs. In v1.36 the defaults are containerLogMaxSize of 10Mi and containerLogMaxFiles of 5. Logs that applications write to their own files are not rotated by kubelet.

Typical scenarios:

  • Debug logging left on in production
  • A crash-looping service producing repetitive errors
  • A log collector holding rotated files open

Container images and runtime artifacts

Large layers, frequent deployments, writable container layers, and incomplete garbage collection all consume image filesystem space while active workloads look normal.

Storage paths differ by runtime:

  • containerd: /var/lib/containerd
  • CRI-O: /var/lib/containers/storage
  • Docker Engine: /var/lib/docker for the classic store, or /var/lib/containerd with the containerd image store

Confirm the active runtime before deleting anything. Kubernetes removed dockershim in v1.24, so don't assume Docker paths on modern nodes.

Ephemeral storage and local workload data

Disk-backed emptyDir volumes, temp files, package caches, spill files, and build artifacts draw from node ephemeral storage. A persistent volume on external storage is not the same thing. A local PersistentVolume is a separate concept too.

Missing requests and limits let one pod or job use more local space than the scheduler assumed. An emptyDir sizeLimit helps, but another consumer can still exhaust shared storage first.

Inodes, filesystem layout, and deleted files

Many small files can exhaust inodes while byte usage looks fine. Separate nodefs and imagefs mounts can also hide trouble: one path looks healthy while another crosses a signal.

df and du may not reconcile for several reasons:

  • A process holds a deleted file open, so space isn't freed
  • Mount changes alter what you can see
  • Kubelet's directory scans miss deleted-but-open files, a blind spot the ephemeral storage docs acknowledge

Check df -i and look for processes holding deleted files open before dismissing an alert.

What Happens If Kubernetes Disk Pressure Is Ignored

Kubelet tries node-level reclamation first. With a shared nodefs, it garbage-collects dead pods and containers, then unused images. If that isn't enough, it evicts end-user pods.

Kubernetes DiskPressure reclamation and pod eviction response flow

  • Soft thresholds wait for a configured grace period.
  • Hard thresholds allow immediate termination.

The control plane also adds a node.kubernetes.io/disk-pressure:NoSchedule taint, so new pods avoid the node. Reclaiming images doesn't necessarily free the filesystem holding logs, so the layout matters.

Workload impact

  • Pods end up Failed with reason Evicted
  • Scheduling and image pulls fail
  • Restarts repeat and performance degrades
  • Logging components may be unable to write, so you lose observability when you need it most

When byte pressure triggers eviction, order depends on whether usage exceeds requests, then pod priority, then usage relative to requests. QoS class is not the direct criterion.

Wider consequences

Evicted workloads move to other nodes, which then fill faster. Recovery actions become disruptive. A single-node capacity issue becomes a cluster stability incident.

An upstream report on Kubernetes 1.30 (issue #127548) even describes other pods being evicted before the hostPath pod that caused the pressure. The node does not always remove the workload that filled the disk.

Warning Signs You're About to Experience Kubernetes Disk Pressure

No single percentage alert catches everything. Combine node conditions, filesystem trends, workload behavior, and events.

DiskPressure=True or eviction-related node events

Start here:

kubectl get nodes
kubectl describe node <node>

Look for these messages:

  • KubeletHasDiskPressure / kubelet has disk pressure
  • NodeHasDiskPressure on transition, NodeHasNoDiskPressure on recovery
  • EvictionThresholdMet: Attempting to reclaim ...
  • The node was low on resource: ephemeral-storage

These pod messages signal limit evictions, not necessarily node-wide pressure:

  • Container ... exceeded its local ephemeral storage limit
  • Pod ephemeral local storage usage exceeds the total limit of containers
  • Usage of EmptyDir volume ... exceeds the limit

Free space or inode availability is falling unexpectedly

Work through this sequence:

  1. Run df -h and df -i on every mount, including separate nodefs and imagefs.
  2. Use du on log, kubelet, runtime, and temp directories.
  3. Check for deleted files still held open.
  4. Query the kubelet Summary API: kubectl get --raw "/api/v1/nodes/<node>/proxy/stats/summary".

Kubernetes DiskPressure diagnostic sequence using filesystem and kubelet checks

The Summary API reports pod ephemeral-storage, container rootfs and logs, availableBytes, and inodesFree. That lets you attribute usage to a pod instead of only the node.

Identify the consumer before deleting anything. Preserve logs required by retention or incident policy.

Pods become Evicted, Pending, or repeatedly rescheduled

Correlate pod status and events with the affected node. Inspect these areas to find the trigger:

  • Pod description and recent events
  • Ephemeral-storage requests and limits
  • emptyDir usage and log growth
  • Image churn and recent releases

How to Prevent Kubernetes Disk Pressure

Treat prevention as layers: limit what workloads write locally, control what the node retains, observe growth, and keep recovery headroom.

Control container log growth

  • Set containerLogMaxSize and containerLogMaxFiles for your Kubernetes version and runtime
  • Lower unnecessary verbosity
  • Ship required logs to a central destination with a deliberate retention policy

Do this at cluster buildout, after runtime changes, and whenever a new logging pipeline or noisy workload arrives. Avoid ad hoc cleanup DaemonSets and blind truncation of active logs. Ad hoc cleanup can break collectors, permissions, or retention requirements.

Govern ephemeral storage at the pod and namespace level

  • Set realistic ephemeral-storage requests and limits
  • Use emptyDir sizeLimit where it fits
  • Apply ResourceQuota (requests.ephemeral-storage, limits.ephemeral-storage) and LimitRange defaults per namespace
  • Reserve space with kubeReserved and systemReserved

Base values on measured behavior, not copied examples. Validate them for batch, AI, CI/CD, and data-processing jobs.

Put storage-heavy workloads on dedicated node pools using taints, tolerations, labels, and node affinity. A toleration alone doesn't pin pods to a pool.

Manage images, runtime data, and node filesystem capacity

Defaults in v1.36 for image garbage collection:

Setting Default
imageGCHighThresholdPercent 85
imageGCLowThresholdPercent 80
imageMinimumGCAge 2m
imageMaximumGCAge 0s (disabled)

Keep images small, trim unneeded layers, and watch deployment churn. Verify which filesystem kubelet monitors before changing mounts or attaching disks.

Bigger disks add headroom. They don't fix runaway logs, temp files, image retention, or inode exhaustion.

Monitor storage trends and automate a safe response

Alert on:

  • The DiskPressure condition
  • Available bytes and inodes
  • Log growth and image filesystem usage
  • Pod ephemeral-storage consumption
  • The rate at which free space is falling

Useful metric sources:

  • kubelet Summary API
  • node_exporter (node_filesystem_avail_bytes, node_filesystem_files_free)
  • cAdvisor (container_fs_usage_bytes, container_fs_inodes_free)

Set thresholds from workload behavior and recovery time, not a universal number.

Build dashboards that name the node, namespace, pod, and deployment behind abnormal growth. Correlate growth spikes with releases, restarts, and batch jobs.

Document a safe response path:

  1. Confirm the alert and affected scope
  2. Clean up safely
  3. Cordon or drain as needed
  4. Reschedule workloads
  5. Replace or expand the node
  6. Run a root-cause review

Six-step Kubernetes DiskPressure safe response and recovery workflow

Tips for Long-Term Prevention and Control

  • Routine checks: Review nodefs, imagefs, inode, log, image, and ephemeral-storage trends on a schedule. Test alerts against realistic growth and recovery.
  • Operator standards: Train teams on persistent vs. node-local ephemeral storage, require production resource policies, and ban unsafe manual deletion.
  • Documentation: Keep a version-aware runbook covering diagnostics, approved cleanup, retention rules, escalation owners, and kubelet or filesystem rollback steps.
  • Capacity intelligence: For cloud block storage on nodes and stateful workloads, use Lucidity for real-time volume visibility across AWS, Azure, and Google Cloud, and AutoScaler to expand and shrink disks with zero downtime. It complements local log, image, inode, and ephemeral-storage controls; it doesn't replace them.

Conclusion

DiskPressure has identifiable causes: logs, images, ephemeral data, inode exhaustion, and layout problems. Checking one directory leads to the wrong diagnosis.

Log governance, ephemeral-storage policy, image lifecycle management, proactive monitoring, and a tested runbook prevent most avoidable evictions and scheduling failures.

Frequently Asked Questions

What does DiskPressure=True mean in Kubernetes?

Kubelet detected that available bytes or inodes on a monitored node filesystem crossed an eviction threshold. It may block scheduling and evict pods to protect the node. The disk need not be literally full.

Why does Kubernetes show disk pressure when df -h shows free space?

Another nodefs or imagefs mount may be the pressured one. Inodes may be exhausted, or a process may hold deleted files open. Check df -i and the relevant filesystem.

Which filesystems does kubelet monitor for disk pressure?

Kubelet monitors nodefs, imagefs, and containerfs where supported, discovered through the runtime. It ignores unrelated mounts. Exact paths depend on your distribution and runtime.

How do I find which pod is consuming ephemeral storage?

Query the kubelet Summary API for pod ephemeral-storage and container rootfs and logs. Then use kubectl describe pod and node events. Inspect emptyDir, temp files, and runtime directories on the node.

How can I prevent container logs from filling a Kubernetes node?

Configure kubelet log rotation, keep application log levels sensible, and ship logs centrally with retention controls. Verify your collector doesn't hold rotated files open. Alert on abnormal log growth.

Does increasing node disk size fix Kubernetes disk pressure?

It adds recovery headroom, nothing more. Unbounded logs, image buildup, temp-file growth, inode exhaustion, and bad filesystem configuration will keep consuming the extra space.