Kubernetes PVC Sizing: Avoid PVC Out of Space and Expand

Introduction

A PVC can show Bound while the filesystem inside the mounted volume is nearly full. Kubernetes reports a healthy claim. Meanwhile, the database throws write errors, a batch job stalls, or the app goes down.

This guide shows platform teams how to size PVCs from real usage and growth, monitor capacity before an incident, confirm that expansion is supported, and increase storage without deleting the claim. It also covers troubleshooting and a fallback migration path.

Key Takeaways

  • Size PVCs from observed growth and expansion lead time, not a generic guess.
  • Monitor bytes and inodes separately from PVC/PV object state.
  • Expand only after confirming StorageClass, CSI driver, quota, and provider support.
  • Plan backups, StatefulSet replicas, and application-level resize steps as part of the operation.

Understand Kubernetes PVC sizing fundamentals

How the storage objects relate

A PersistentVolumeClaim (PVC) is the application-facing request. A PersistentVolume (PV) is the provisioned storage, and a StorageClass defines how storage gets created. A CSI driver connects Kubernetes to the backend, and the kubelet mounts the result as a filesystem.

spec.resources.requests.storage states what the workload asks for. The PV and the cloud volume behind it represent what was provisioned. Pods and StatefulSets should reference the PVC, not a specific PV.

How binding works

Kubernetes binds a PVC to an existing PV or provisions one dynamically. Matching checks:

  • Requested capacity and access mode
  • storageClassName and volumeMode
  • Topology constraints
  • Namespace storage quotas

A static PV can be larger than the request. A claim stays Pending when nothing compatible exists. Note that a nonempty PVC selector prevents dynamic provisioning.

StorageClass settings that affect expansion

Check these fields before you size anything:

  • provisioner: which CSI plugin handles the volume
  • allowVolumeExpansion: must be true for growth
  • volumeBindingMode: Immediate by default; WaitForFirstConsumer waits for Pod scheduling
  • reclaimPolicy: Delete by default, or Retain
  • Filesystem type: driver-specific, such as csi.storage.k8s.io/fstype: xfs under parameters
kubectl get storageclass
kubectl get storageclass <name> -o yaml

Confirm behavior against your Kubernetes version, CSI driver, and cloud provider documentation before relying on version-specific features.

Capacity is not performance

More GiB does not automatically mean more IOPS, throughput, or a different access mode. On AWS, gp2 baseline IOPS scales with size, while gp3 performance is provisioned independently. Azure Premium SSD v2 and Google Hyperdisk work similarly.

Volume mode matters too. Filesystem (the default) mounts a directory. Block exposes a raw device, so the application must handle its own resize.

Inspect before you resize

kubectl get pvc -n <ns>
kubectl describe pvc <name> -n <ns>
kubectl get pv <pv-name> -o yaml
kubectl get pvc <name> -n <ns> -o jsonpath='{.spec.resources.requests.storage} {.status.capacity.storage} {.spec.storageClassName} {.status.phase}'

Capture the request, bound PV, StorageClass, phase, and capacity. Never edit PV capacity by hand to simulate expansion. The supported workflow starts with the PVC request.

Size a PVC before it runs out of space

Establish a baseline

Compare what Kubernetes reports with what the filesystem says:

kubectl exec <pod> -n <ns> -- df -h /data
kubectl exec <pod> -n <ns> -- df -i /data
kubectl exec <pod> -n <ns> -- du -sh /data/*

"Full" can mean several things:

  • Bytes exhausted
  • Inodes exhausted (Red Hat documents inode exhaustion with bytes still free)
  • Reserved blocks (ext4's mke2fs -m defaults to 5% for the superuser)
  • Deleted files still held open by a process
  • An application-level quota
  • A hidden mount path, where the app writes somewhere other than the volume

Build a growth model

Skip arbitrary sizes. Use this formula:

Required size = (current data + growth rate × planning window) × amplification factor + headroom

Example (sample values):

  • Current data: 400 GiB
  • Growth: 8 GiB/day, 60-day planning window = 480 GiB
  • Compaction or rewrite amplification: 1.3×
  • (400 + 480) × 1.3 = 1,144 GiB before headroom

PVC sizing formula with growth amplification and headroom example

Also factor in:

  • Historical growth and planned traffic
  • Bursty writes and temp files
  • Transaction logs and retention policies
  • Per-replica data

Define headroom

Headroom must cover growth plus the time to detect, approve, provision, expand, and validate. A volume growing 8 GiB/day with a two-day approval cycle needs at least 16 GiB free, before any safety margin.

Set a policy with both a minimum free byte count and a minimum free percentage. Tune each to workload behavior and expansion lead time.

Match the PVC to the workload

  • Databases: budget for transaction logs, compaction, and snapshots
  • Log-heavy apps: size to retention policy
  • AI and data jobs: account for temp files and intermediate datasets
  • Caches: keep small and rebuildable
  • StatefulSets: each replica gets its own PVC from volumeClaimTemplates, so size per claim, not only in aggregate

Treat IOPS, throughput, latency, access mode, snapshots, and backup destinations as separate requirements from capacity.

Add guardrails

Namespace ResourceQuota caps aggregate requests.storage and claim count, with per-class keys available. A LimitRange sets PVC min and max sizes. Add ownership labels and GitOps-managed manifests, and review large requests.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: orders-db-data
  labels:
    owner: payments-team
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: gp3-expandable
  resources:
    requests:
      storage: 200Gi

Over-provisioning raises spend and doesn't replace monitoring.

Monitor PVC capacity and predict out-of-space risk

What to watch

Track these signals to catch capacity risk early:

  • Filesystem used and available bytes
  • Inode usage
  • PVC/PV status and mount health
  • Backend health, IOPS, throughput, and latency
  • Application write errors

The kubelet exposes these metrics (labeled ALPHA in the Kubernetes metrics reference):

  • kubelet_volume_stats_used_bytes
  • kubelet_volume_stats_available_bytes
  • kubelet_volume_stats_capacity_bytes
  • kubelet_volume_stats_inodes, _inodes_free, _inodes_used

Verify collection for your distribution. GKE's managed package needs 1.29.3-gke.1093000 or later, and AKS Managed Prometheus includes these by default. The EKS Prometheus docs don't state default PVC byte collection, so confirm it in your setup.

Alert on current and future risk

Pair threshold alerts with forecasts. Percentage-only alerts miss small but critical volumes. Byte-only alerts miss huge volumes that grow fast. Use both.

The kubernetes-mixin warning rule combines under 15% available with predict_linear(kubelet_volume_stats_available_bytes[6h], 4*24*3600) < 0 for one hour. Its critical rule fires at under 3% available.

PVC capacity alert thresholds and four-hour forecast rule diagram

Also alert on:

  • Inode exhaustion
  • Rapid growth
  • Failed writes
  • PVC Pending
  • Resize conditions

Set thresholds from your own policy, not these defaults.

Alert response runbook

When an alert fires:

  1. Confirm the namespace, PVC, PV, Pod, mount path, and consuming process.
  2. Correlate metrics with kubectl describe pvc, Pod events, app logs, and provider events.
  3. Decide: cleanup, expansion, workload fix, or migration.
  4. Document the outcome.

Don't delete files until you know what's consuming space.

Expand a Kubernetes PVC safely

Pre-expansion checks

  • StorageClass has allowVolumeExpansion: true
  • CSI driver and backend support it
  • Namespace quota has room
  • Target size is within provider limits (EBS gp3 reaches 64 TiB, gp2 16 TiB)
  • A backup or snapshot exists, especially for databases

On AWS, a prior modification must complete first, and AWS allows four modifications per rolling 24 hours. Azure may require detaching disks in some configurations, such as crossing the 4 TiB boundary.

Increase the request

kubectl patch pvc orders-db-data -n payments \
  -p '{"spec":{"resources":{"requests":{"storage":"400Gi"}}}}'

Change only spec.resources.requests.storage. Expansion is increase-only:

  • Don't shrink the request
  • Don't edit PV capacity
  • Don't delete and recreate the PVC

With GitOps, update the source manifest too, or reconciliation reverts it.

Watch the resize

kubectl get pvc orders-db-data -n payments -w
kubectl describe pvc orders-db-data -n payments

Watch for these conditions:

  • Resizing: the backend is growing the volume
  • FileSystemResizePending: the backend finished; the node still has to grow the filesystem

Kubernetes PVC expansion lifecycle from backend resize to filesystem growth

That second step needs a Pod using the PVC. Online filesystem expansion is stable since v1.24 for XFS, ext3, and ext4.

Since v1.34, failed expansions can recover by retrying with a smaller request above current capacity. Look for ControllerResizeError and NodeResizeError.

Validate inside the workload

kubectl exec <pod> -n payments -- df -h /data

Confirm the new size, run a safe write test, and check that the application sees the space. Online expansion depends on the driver, filesystem, version, and workload.

If the driver or filesystem docs call for a remount or restart, schedule it. Online expansion is not zero-downtime in every setup.

Stateful and application caveats

  • A database may need its own command or config change to use new space
  • StatefulSet templates can't resize in place; patch each existing PVC
  • Snapshot restores can't go below restoreSize
  • Expansion doesn't delete data or fix a runaway writer
  • To shrink, migrate or export and restore

When automation helps

At scale, manual expansion becomes ticket churn. Lucidity is built for enterprises managing cloud block storage on AWS, Azure, and Google Cloud.

It gives real-time visibility into every volume, and AutoScaler expands, shrinks, and optimizes disks with zero downtime. AutoScaler has eliminated 517,537 manual SRE tasks.

Confirm your Kubernetes, CSI, and cloud-provider integration before assuming direct PVC automation. When that fit is right, you cut manual storage operations and over-provisioning.

Troubleshoot PVC out-of-space and expansion problems

Bound but no space

  1. Compare df -h with du. A gap suggests deleted-but-open files.
  2. Run df -i for inode exhaustion.
  3. Find deleted-open files with lsof +L1 (Red Hat documents this).
  4. Check logs and temp directories.
  5. Confirm the Pod uses the expected mount path.
  6. Check application quotas.

Don't delete database files, logs, or temp data without understanding retention and recovery.

Pending claim or stuck expansion

Check these signals before you change capacity:

  • PVC and Pod events
  • Requested size and StorageClass name
  • Access mode and topology
  • ResourceQuota limits
  • Provisioner health and backend capacity

Common failure signals:

  • Quota violation: rejected with 403 Forbidden
  • Unsatisfied claim: admitted but waiting; WaitForFirstConsumer delays binding on purpose
  • Resize failure: look for ControllerResizeError
  • Filesystem pending: FileSystemResizePending
  • Provider limit: check cloud events or console

Fallback migration

When in-place expansion isn't available:

  1. Create a larger compatible PVC.
  2. Copy data in an application-consistent way, or restore from a snapshot (CSI cloning needs destination capacity at least equal to the source).
  3. Validate permissions and ownership.
  4. Update the workload and cut over in a controlled window.
  5. Confirm the old PV's reclaim policy before delete, then clean up so orphaned cloud storage does not keep billing.

Five-step fallback PVC migration process for larger Kubernetes storage

Frequently Asked Questions

What is the difference between a persistent volume and a persistent volume claim in Kubernetes?

A PV is the provisioned storage resource. A PVC is a workload's request for storage, which Kubernetes binds to a compatible PV.

What is a persistent volume?

A PV is a Kubernetes storage resource backed by local, network, or cloud storage. It can outlive any individual Pod.

What is the difference between ephemeral and persistent storage?

Ephemeral storage follows the Pod's lifecycle. Persistent storage remains available across Pod replacement or restart.

Does deleting a PVC delete the PV?

It depends on the reclaim policy. Delete removes the PV and backing storage, while Retain leaves the PV and data for manual reclamation. Verify the underlying asset before deleting.

How do I increase the size of a Kubernetes PVC?

Verify expansion support, then raise spec.resources.requests.storage. Watch PVC events and resize conditions, then validate with df -h inside the Pod.

Why is my Kubernetes PVC still full after expansion?

Check whether the filesystem resize finished (FileSystemResizePending), then look for inode exhaustion, deleted-but-open files, application quotas, or a wrong mount path. Use df, events, and CSI driver status to confirm.