RWX volumes on VMstore: the CSI call sequence, and the five places it strands

Almost every RWX troubleshooting session goes wrong in the same way: the engineer reasons about the volume as though it belonged to the pod. It does not. Under ReadWriteMany the staged mount belongs to the node, is shared by every pod on that node, and is reference counted. Nearly every confusing RWX failure follows from that one fact.

This is the first of a series on troubleshooting Kubernetes and vSphere workloads backed by a Tintri VMstore over the CSI driver, with every scenario in RWX. Start here, because the later posts assume you can read the call sequence.

from kubernetes import client, config

config.load_kube_config()                    # or load_incluster_config()

core    = client.CoreV1Api()
storage = client.StorageV1Api()

core.list_persistent_volume_claim_for_all_namespaces()  # GET /api/v1/persistentvolumeclaims
core.list_persistent_volume()                           # GET /api/v1/persistentvolumes
core.list_pod_for_all_namespaces()                      # GET /api/v1/pods
storage.list_volume_attachment()                        # GET /apis/storage.k8s.io/v1/volumeattachments

# every one of these is a plain GET; the script never writes anything

# 1. do you have the read access the triage script needs?
for r in persistentvolumeclaims persistentvolumes pods \
         volumeattachments storageclasses csinodes csidrivers; do
  printf '%-24s %s\n' "$r" "$(kubectl auth can-i list $r --all-namespaces)"
done

# 2. is the driver registered as a CSIDriver object, and does it need attach?
kubectl get csidriver -o custom-columns=\
NAME:.metadata.name,ATTACH:.spec.attachRequired,PODINFO:.spec.podInfoOnMount,\
MODES:.spec.volumeLifecycleModes

# 3. did it register on EVERY node? a missing row here explains
#    'works on some nodes, not others' without any log reading
kubectl get csinode -o custom-columns=\
NODE:.metadata.name,DRIVERS:.spec.drivers[*].name
kubectl get nodes --no-headers | wc -l   # compare the counts

# 4. which controller replica currently holds leadership?
kubectl -n <driver-ns> get lease -o custom-columns=\
NAME:.metadata.name,HOLDER:.spec.holderIdentity,RENEWED:.spec.renewTime

ILLUSTRATIVE SHAPES, not captured output. Match against your own cluster.

STRAND 1  provisioning never happened
  pvc  STATUS=Pending    VOL=<empty>
  pv   (none)
  pod  Pending, event: waiting for first consumer / or a provisioner error

STRAND 2  provisioned, cannot mount on this node
  pvc  STATUS=Bound      VOL=pvc-8f2c...
  pv   STATUS=Bound      HANDLE=<array-side id>
  pod  Pending  containerStatuses[].state.waiting.reason=ContainerCreating
  event: MountVolume.SetUp failed ...

STRAND 3  orphaned stage reference
  pvc  STATUS=Bound
  pod  no consumer pods on node-04 at all
  volumeattachment  NODE=node-04  ATTACHED=true      <-- the tell
  on node-04: globalmount still mounted, zero pod bind mounts

STRAND 4  mounted and running, I/O failing
  pvc  STATUS=Bound
  pod  Running, restartCount climbing or app logging EIO / ESTALE
  nothing in CSI is wrong; the fault is in the NFS session

STRAND 5  unpublish blocked
  pod  metadata.deletionTimestamp set (non-null)
       metadata.finalizers still populated
  driver log: NodeUnpublishVolume ... device or resource busy

01. The call sequence, and where RWX diverges from RWO

CONTROLLER PLANE                          NODE PLANE
(one leader, cluster-wide)                (one daemonset pod per node)

  CreateVolume
    creates the export on the VMstore
    idempotent on the volume name
         |
         v
  ControllerPublishVolume
    RWO : real attach, one node only
    RWX : usually a no-op for NFS,
          the export is already reachable
          from every node on the storage VLAN
         |
         +---------------------------------> NodeStageVolume
                                               mounts the export ONCE per node at
                                               /var/lib/kubelet/plugins/kubernetes.io/
                                                 csi/<driver>/<sha>/globalmount
                                               REFERENCE COUNTED across pods
                                                     |
                                                     v
                                             NodePublishVolume
                                               bind mount globalmount -> pod dir
                                               called once PER POD
                                                     |
                                             ... pod runs ...
                                                     |
                                             NodeUnpublishVolume   (per pod)
                                                     |
                                             NodeUnstageVolume     (only when the
                                                                    LAST pod on that
                                                                    node releases it)

The consequence worth internalising: a single stale reference on a node keeps the global mount alive forever. NodeUnstageVolume is never called, the lease is never released, and the next scheduling decision that touches that volume behaves in ways that make no sense if you are thinking per pod.

02. Establish which layer owns the problem, in thirty seconds

Before reading any driver logs, work out which of the four phases you are stuck in. The object states tell you unambiguously.

# 1. did provisioning happen at all?
kubectl get pvc -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,\
MODE:.spec.accessModes[0],SC:.spec.storageClassName,VOL:.spec.volumeName

# 2. does the PV exist and what is its reclaim state?
kubectl get pv -o custom-columns=\
NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.name,\
POLICY:.spec.persistentVolumeReclaimPolicy,\
HANDLE:.spec.csi.volumeHandle

# 3. controller-side attachment objects
kubectl get volumeattachment -o custom-columns=\
NAME:.metadata.name,ATTACHED:.status.attached,NODE:.spec.nodeName,\
PV:.spec.source.persistentVolumeName

# 4. what is the pod actually waiting on?
kubectl describe pod <pod> -n <ns> | sed -n '/Events:/,$p'

03. Strand 1: PVC Pending on an RWX StorageClass

Symptom. PVC sits in Pending. Events may say nothing at all, which is itself diagnostic.

The trap specific to RWX. If your StorageClass uses volumeBindingMode: WaitForFirstConsumer, the PVC is supposed to stay Pending until a pod is scheduled. Engineers routinely spend an hour debugging a provisioner that has not been asked to do anything yet.

# is it waiting deliberately?
kubectl get sc <class> -o jsonpath='{.volumeBindingMode}{"\n"}'
# WaitForFirstConsumer -> Pending with no consumer pod is CORRECT

# is the provisioner even watching this class?
kubectl get sc <class> -o jsonpath='{.provisioner}{"\n"}'
kubectl get csidriver

# the provisioner's own view
kubectl -n <driver-ns> logs deploy/<controller> -c csi-provisioner --tail=200 \
  | grep -Ei 'failed|error|retry|provision'

# leader election: a controller that is not the leader logs nothing useful
kubectl -n <driver-ns> get lease
kubectl -n <driver-ns> logs deploy/<controller> -c csi-provisioner \
  | grep -i 'became leader'

Causes, in the order they actually occur:

1. WaitForFirstConsumer with no schedulable pod. Not a storage problem. Look at why the pod will not schedule.
2. StorageClass parameters that do not match the array. A parameter naming the wrong appliance, pool or zone. The provisioner reports the array’s rejection, which is usually clear once you find it.
3. Parameter name drift between driver versions. A key renamed between releases is silently ignored, so the driver falls back to a default and provisions somewhere you did not intend. This one does not fail, which makes it worse. Confirm the parameter names against the version you are actually running, not the version the runbook was written for.
4. Credentials expired. The controller’s secret holds credentials for the array. An expired password or certificate surfaces as an authentication failure buried in the driver container, not the provisioner sidecar.
5. No leader. If leader election is broken, every controller replica is passive and nothing provisions. Check the lease.

04. Strand 2: pod stuck in ContainerCreating

The PV exists and is Bound, so provisioning worked. The failure is on the node. Go to the node daemonset, not the controller.

NODE=$(kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}')
DS_POD=$(kubectl -n <driver-ns> get pod -l app=<node-daemonset-label> \
           --field-selector spec.nodeName=$NODE -o name | head -1)

# the driver's own view of stage and publish
kubectl -n <driver-ns> logs $DS_POD -c <driver-container> --tail=300 \
  | grep -Ei 'nodestage|nodepublish|mount|denied|timeout|stale'

# kubelet's view, which often carries the real mount error
kubectl get events -n <ns> --field-selector involvedObject.name=<pod> \
  --sort-by=.lastTimestamp

If the logs point at the mount itself, inspect the node directly. This is the step most people skip, and it is where the answer usually is.

# on the node, or via a debug pod with hostPID and host mounts

# is the global mount present, and what is it?
findmnt -t nfs,nfs4 -o TARGET,SOURCE,OPTIONS \
  | grep -i 'kubernetes.io/csi'

# how many pods reference this staged mount right now
SHA=<the sha in the globalmount path>
findmnt -o TARGET | grep -c "$SHA"

# can this node even reach the export, independent of Kubernetes
showmount -e <vmstore-data-ip>
rpcinfo -t <vmstore-data-ip> nfs 3

# prove the path at full frame size before blaming the driver
ping -M do -s 8972 -c 3 <vmstore-data-ip>

05. Strand 3: orphaned stage references, the RWX-specific failure

This is the one that does not exist in RWO, and the one that produces the strangest symptoms.

Mechanism. NodeUnstageVolume fires only when the last pod on a node releases the volume. If a node is lost ungracefully, or a pod is force deleted while its bind mount is still live, the reference is never decremented. The global mount stays. The driver still believes that node holds the volume.

Why it escalates. Every subsequent reconcile retries the unstage, fails, and backs off. Accumulate enough of these and the driver’s rate limiter saturates. At that point unrelated volume operations start timing out, and the reported symptom is “provisioning is slow” on a completely different namespace. The causal link is invisible unless you go looking for stale references.

#!/usr/bin/env bash
# rwx-orphan-scan.sh
# Finds staged RWX mounts on a node with no live pod referencing them.
# Read only. Run on each node, or from a privileged debug daemonset.

set -uo pipefail
CSI_ROOT=/var/lib/kubelet/plugins/kubernetes.io/csi
PODS_ROOT=/var/lib/kubelet/pods

printf '%-46s %-9s %s\n' VOLUME_SHA REFCOUNT VERDICT
printf '%s\n' "$(printf '=%.0s' {1..78})"

orphans=0
for gm in "$CSI_ROOT"/*/*/globalmount; do
  [ -d "$gm" ] || continue
  sha=$(basename "$(dirname "$gm")")

  # is it actually mounted, or just a leftover directory?
  if ! mountpoint -q "$gm"; then
    printf '%-46s %-9s %s\n' "${sha:0:44}" "-" "STALE DIR, not mounted"
    orphans=$((orphans+1))
    continue
  fi

  # count live per-pod bind mounts referencing this volume
  refs=$(grep -c "$sha" /proc/self/mountinfo || true)
  # subtract the globalmount itself
  refs=$((refs - 1))

  # count pod directories that still reference it on disk
  podrefs=$(find "$PODS_ROOT" -maxdepth 5 -type d -name "*$sha*" 2>/dev/null | wc -l)

  if [ "$refs" -le 0 ] && [ "$podrefs" -eq 0 ]; then
    printf '%-46s %-9s %s\n' "${sha:0:44}" "$refs" "ORPHANED, no pod refs"
    orphans=$((orphans+1))
  else
    printf '%-46s %-9s %s\n' "${sha:0:44}" "$refs" "in use ($podrefs pod dirs)"
  fi
done

echo
echo "orphaned staged volumes on $(hostname): $orphans"
[ "$orphans" -eq 0 ] || exit 2

Recovery, in this order. Do not start by unmounting.

1. Confirm no pod is genuinely using it. The script tells you; verify against kubectl get pods -o wide --field-selector spec.nodeName=<node>. An unmount against a live workload corrupts data.
2. Cordon the node. You do not want a new pod landing mid-recovery.
3. Restart the node driver pod on that node first. This clears in-memory operation locks and frequently resolves the whole thing without touching mounts.
4. Only if it persists, unmount the specific globalmount path, then remove the empty directory. Use a lazy unmount only as a last resort, and understand it defers rather than solves.
5. Uncordon and verify a new pod on that node can mount the same volume.

Prevention beats recovery here. Force deleting a pod with --grace-period=0 --force while it holds an RWX volume is the single most reliable way to create an orphan. It removes the API object without waiting for NodeUnpublish, so the reference on the node is never released.

06. Strand 4: mounted, running, and returning I/O errors

CSI has done its job. The problem is the NFS session underneath, and RWX makes it more likely because many clients share one export.

The last row deserves saying plainly. RWX gives you concurrent write access. It does not give you coordination. If two pods write the same file without application-level locking, you will get inconsistent data, and no amount of driver or array tuning will fix it. NFS advisory locking works but requires the application to actually use it. Most workloads people put on RWX expect a shared filesystem to behave like a database, and it does not.

# confirm what the client negotiated, per node
findmnt -t nfs,nfs4 -o TARGET,SOURCE,OPTIONS | grep csi
# check vers=, hard/soft, actimeo, lock/nolock, and that they MATCH across nodes

# per-operation error counters on this client
nfsstat -c

# lock state as the server sees it
rpcinfo -p <vmstore-data-ip> | grep -Ei 'nlockmgr|status'

# who holds the file open inside the pod
kubectl exec -n <ns> <pod> -- sh -c 'ls -l /proc/*/fd 2>/dev/null | grep <mount>'

Mount options differing between nodes is a real and under-diagnosed cause. One node mounting soft while the rest mount hard produces I/O errors on that node alone during a brief array pause, while every other node simply waits and recovers. The symptom looks like a node problem; the cause is a mount option.

07. Strand 5: pod stuck Terminating

The pod will not go away. Deleting it again does nothing, because it is already deleted as far as the API is concerned and is waiting on a finalizer.

# what is it actually waiting on
kubectl get pod <pod> -n <ns> -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl get pod <pod> -n <ns> -o jsonpath='{.metadata.deletionTimestamp}{"\n"}'

# is the unpublish being attempted and failing?
kubectl -n <driver-ns> logs $DS_POD -c <driver-container> --tail=200 \
  | grep -Ei 'nodeunpublish|nodeunstage|device or resource busy'

# on the node: what still holds the bind mount open
fuser -vm /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<pv>/mount
lsof +D /var/lib/kubelet/pods/<pod-uid>/volumes 2>/dev/null | head

Almost always a process inside the container still holds a file open on the mount, and unpublish correctly refuses. The fix is to let the process exit, not to force the object away. Removing the finalizer by hand deletes the API object and leaves the mount stranded on the node, which converts a five minute wait into the orphan problem in section 05.

08. A triage script that answers the first question for you

#!/usr/bin/env python3
"""rwx_triage.py - classify every RWX volume in the cluster by failure phase.

Read only. Uses the Kubernetes API alone, so it runs from anywhere with
a kubeconfig and needs no node access.
"""

import argparse
import sys
from collections import defaultdict

from kubernetes import client, config


def classify(pvc, pv, pods, attachments):
    """Return (phase, detail) for one RWX claim."""
    name = f"{pvc.metadata.namespace}/{pvc.metadata.name}"

    if pvc.status.phase != "Bound":
        consumers = [p for p in pods if uses(p, pvc)]
        if not consumers:
            return ("PENDING_NO_CONSUMER",
                    "unbound with no pod; correct if WaitForFirstConsumer")
        return ("PROVISION_FAILED",
                f"unbound but {len(consumers)} pod(s) waiting")

    if pv is None:
        return ("BOUND_NO_PV", "claim is Bound but the PV is missing")

    consumers = [p for p in pods if uses(p, pvc)]
    nodes = {p.spec.node_name for p in consumers if p.spec.node_name}

    stuck_creating = [p for p in consumers
                      if p.status.phase == "Pending"
                      and any(cs.state.waiting and
                              cs.state.waiting.reason == "ContainerCreating"
                              for cs in (p.status.container_statuses or []))]
    if stuck_creating:
        return ("MOUNT_FAILED",
                "stuck ContainerCreating on " +
                ", ".join(sorted(p.spec.node_name or "?"
                                 for p in stuck_creating)))

    terminating = [p for p in consumers if p.metadata.deletion_timestamp]
    if terminating:
        return ("UNPUBLISH_STUCK",
                "terminating: " +
                ", ".join(p.metadata.name for p in terminating))

    # attachments naming nodes that host no consumer pod
    handle = pv.spec.csi.volume_handle if pv.spec.csi else None
    stale = []
    for va in attachments:
        if va.spec.source.persistent_volume_name != pv.metadata.name:
            continue
        if va.status and va.status.attached and va.spec.node_name not in nodes:
            stale.append(va.spec.node_name)
    if stale:
        return ("STALE_ATTACHMENT",
                "attached on nodes with no consumer: " + ", ".join(sorted(stale)))

    return ("OK", f"{len(consumers)} pod(s) across {len(nodes)} node(s)")


def uses(pod, pvc):
    for v in pod.spec.volumes or []:
        c = v.persistent_volume_claim
        if c and c.claim_name == pvc.metadata.name \
                and pod.metadata.namespace == pvc.metadata.namespace:
            return True
    return False


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--kubeconfig")
    p.add_argument("--driver", help="only claims from this provisioner")
    a = p.parse_args()

    config.load_kube_config(config_file=a.kubeconfig)
    core, storage = client.CoreV1Api(), client.StorageV1Api()

    pvcs = [c for c in core.list_persistent_volume_claim_for_all_namespaces().items
            if "ReadWriteMany" in (c.spec.access_modes or [])]
    pvs = {v.metadata.name: v for v in core.list_persistent_volume().items}
    pods = core.list_pod_for_all_namespaces().items
    attachments = storage.list_volume_attachment().items

    if a.driver:
        pvcs = [c for c in pvcs
                if (pvs.get(c.spec.volume_name) and
                    pvs[c.spec.volume_name].spec.csi and
                    pvs[c.spec.volume_name].spec.csi.driver == a.driver)]

    buckets = defaultdict(list)
    for c in pvcs:
        phase, detail = classify(c, pvs.get(c.spec.volume_name),
                                 pods, attachments)
        buckets[phase].append(
            (f"{c.metadata.namespace}/{c.metadata.name}", detail))

    print(f"RWX claims examined: {len(pvcs)}\n")
    for phase in sorted(buckets, key=lambda k: (k == "OK", k)):
        print(f"[{phase}]")
        for name, detail in sorted(buckets[phase]):
            print(f"    {name:<52} {detail}")
        print()

    bad = sum(len(v) for k, v in buckets.items() if k != "OK")
    print(f"claims needing attention: {bad}")
    return 0 if bad == 0 else 2


if __name__ == "__main__":
    sys.exit(main())

Point it at a cluster and it tells you, per claim, which of the five strands you are in. That is the thirty seconds of triage that decides whether you open the controller logs, the node logs, or the array.

Next in this series: RWX under KubeVirt live migration, where two nodes legitimately hold the same volume at once and the refcount reasoning above has to be extended.

Container names, daemonset labels and StorageClass parameter keys differ by driver build, so substitute your own throughout. The node scripts require privileged host access and are read only by design. Confirm no live workload is using a volume before unmounting anything. Nothing here is official guidance from any vendor.