A pod writes to a volume. Four layers later the bits land on flash. When that write gets slow, four teams open four consoles and each one reports that their layer looks fine. This post builds the whole path deliberately, then gives you the runbook that tells you which layer is lying.
The workload is a stateful one: a PostgreSQL StatefulSet plus a shared artifact volume, running on OpenShift, on vSphere, on Cisco UCS blades, backed by a Tintri VMstore over NFS. That combination is common and it is where layer boundaries hurt most, because Kubernetes has no idea the array exists and the array has no idea the pod exists.
Contents
- The stack, and where each layer can lie to you
- Design decisions, one per layer
- Build layer 1: Cisco UCS
- Build layer 2: Tintri VMstore and NFS
- Build layer 3: ESXi and vSphere
- Build layer 4: OpenShift and the CSI driver
- Validate before you hand it over
- Triage: which layer owns the latency
- Runbook: esxtop
- Runbook: NFS and VMstore
- Runbook: UCS fabric and upstream switches
- Runbook: Kubernetes and CSI
01. The stack, and where each layer can lie to you
POD postgres-0 writes 8 KiB, fsync
| sees: application latency
v
CSI PVC -> VMDK on NFS datastore
| sees: mount state, attach state, nothing about speed
v
GUEST RHCOS worker VM, pvscsi controller
| sees: block latency inside the guest
v
ESXi VMkernel queue -> NFS client -> vmk1
| sees: GAVG, KAVG, queue depth
v
UCS vNIC -> IOM -> Fabric Interconnect -> Nexus
| sees: drops, pause frames, MTU mismatches
v
VMSTORE NFS export, per-VM QoS, flash
sees: array-side latency, per-VM, honestlyEach layer measures a different span of the same write, and each one reports only its own span. The application sees the whole thing. The array sees the last hop. The gap between those two numbers is the entire problem, and it is where every difficult ticket lives.
| Layer | What it measures | How it misleads you |
|---|---|---|
| Pod | End to end application latency | Blames storage for everything, including CPU contention |
| CSI | Attach and mount state only | Reports Healthy while performance is unusable |
| ESXi | GAVG, KAVG, queue depth | On NFS the usual device counters are empty, see the esxtop section |
| UCS | Frames, drops, pause | An MTU mismatch shows as intermittent storage latency, not a network error |
| VMstore | Array-side service time per VM | Looks perfect while the fabric is dropping frames |
02. Design decisions, one per layer
Jumbo frames end to end, or 1500 everywhere
Choose one and enforce it at every hop. A partial jumbo configuration is worse than no jumbo configuration, because small packets succeed and large ones are silently dropped, which presents as intermittent NFS latency rather than as a network fault. If you cannot guarantee 9216 on the fabric interconnect, the IOM, the upstream Nexus and the array port, stay at 1500 and lose a few percent of throughput.
One NFS datastore per workload class, not one giant one
VMstore gives per-VM visibility and QoS regardless of datastore layout, so the usual argument for one large datastore is weaker here. Separate datastores give you separate mount points to unmount, separate heartbeats to fail, and a blast radius you can reason about. Split at least production from non-production.
Dedicated storage vmkernel on its own VLAN
Storage traffic gets its own vmkernel adapter, its own VLAN and its own uplink preference. Sharing a vmk with management means a vCenter task can compete with database writes, and it makes the esxtop network view useless for attribution.
Anti-affinity for the StatefulSet, at both layers
Kubernetes pod anti-affinity spreads replicas across worker nodes. That is worthless if DRS then places all those workers on one ESXi host. You need a vSphere anti-affinity rule for the worker VMs as well. This is the single most commonly missed step in the entire build, and it turns a host failure into a full outage.
Right-size worker vCPU before you tune anything else
Wide worker VMs on a consolidated cluster generate co-stop, and co-stop looks exactly like slow storage from inside the pod. Start narrow and grow. Four vCPU workers that are never scheduled beat twelve vCPU workers that wait.
03. Build layer 1: Cisco UCS
Start at the bottom of the network path. Everything above assumes this is right.
QoS system class and MTU
In UCS Manager, LAN, LAN Cloud, QoS System Class. Set the class you will use for storage to MTU 9216. The fabric must be configured for 9216 even though the vmkernel will be 9000, because the fabric number includes headers.
# verify from the FI CLI
connect nxos a
show queuing interface ethernet 1/1 | include MTU
show interface ethernet 1/1 | include MTU
# confirm the class configuration UCSM pushed
show policy-map type network-qos
# repeat on fabric B
exit
connect nxos b
show queuing interface ethernet 1/1 | include MTUvNIC template and service profile
- Create a vNIC template for storage with MTU 9000, the storage VLAN as native or tagged consistently, and fabric failover disabled if you are using vSphere-level teaming.
- Bind the QoS policy that maps to your jumbo-enabled system class. A vNIC set to MTU 9000 against a class that is still at 1500 will drop large frames.
- Give storage vNICs a deterministic PCI order so vmnic numbering is identical across every blade. Non-deterministic order is why one host in twelve behaves differently.
Check this before moving on. A vNIC at 9000 bound to a QoS class at 1500 is the classic silent failure. Small NFS operations succeed, metadata works, the datastore mounts, and then large reads stall. You will chase this at the array for a week if you skip the verification above.
04. Build layer 2: Tintri VMstore and NFS
- Configure the data IP on the storage VLAN, separate from the admin IP. Never mount a datastore against the admin interface.
- Set the data interface MTU to 9000 to match the vmkernel.
- Restrict the export to the storage subnet only. An export open to the management network is both a security problem and a support problem.
- Register vCenter in the VMstore Hypervisor Manager. This is not optional decoration. Per-VM visibility, per-VM QoS and VM-consistent snapshots all depend on it, and without it the array shows you files rather than VMs.
Set a QoS floor on the database VMs once the platform is live and you have a baseline. Setting a floor before you know normal is guessing, and a floor that is too high starves everything else on the array.
05. Build layer 3: ESXi and vSphere
# create the storage vmkernel, jumbo, on the storage portgroup
esxcli network ip interface add -i vmk1 -p "Storage"
esxcli network ip interface ipv4 set -i vmk1 -I 10.20.30.41 -N 255.255.255.0 -t static
esxcli network ip interface set -i vmk1 -m 9000
# confirm the vSwitch or DVS uplink also carries 9000
esxcli network vswitch standard list | grep -i mtu
esxcli network ip interface list | grep -A2 vmk1Now prove the path before you mount anything. This single command validates MTU across the vNIC, the fabric interconnect, the upstream switch and the array port in one shot.
# 8972 = 9000 minus 20 bytes IP header minus 8 bytes ICMP header
# -d sets do-not-fragment. Without -d the test is meaningless.
vmkping -I vmk1 -s 8972 -d 10.20.30.10
# compare against a small packet to isolate MTU from reachability
vmkping -I vmk1 -s 1472 -d 10.20.30.10Read the result carefully. Small succeeds and large fails means an MTU mismatch somewhere in the path. Both fail means a VLAN, routing or export problem. Both succeed means the path is clean and you can mount. Do not mount until the 8972 test passes, because a datastore on a broken jumbo path will mount fine and fail under load.
# mount the datastore
esxcli storage nfs add -H 10.20.30.10 -s /tintri/ocp-prod-01 -v OCP-PROD-01
# verify
esxcli storage nfs list
esxcli storage nfs param get -v OCP-PROD-01
# heartbeat and volume limits worth knowing
esxcli system settings advanced list -o /NFS/MaxVolumes
esxcli system settings advanced list -o /Net/TcpipHeapSize
esxcli system settings advanced list -o /Net/TcpipHeapMax
esxcli system settings advanced list -o /NFS/HeartbeatFrequency
esxcli system settings advanced list -o /NFS/HeartbeatMaxFailuresIf you are mounting many NFS datastores, raise NFS.MaxVolumes and the TCP/IP heap together. Raising volume count without heap produces mount failures that look like array problems. Check your array vendor’s current guidance for the exact values rather than copying numbers from a blog, including this one.
Anti-affinity for the worker VMs
Connect-VIServer vcenter.lab.local
$workers = Get-VM -Name "ocp-worker-*"
New-DrsRule -Cluster "OCP-PROD" `
-Name "ocp-workers-separate" `
-KeepTogether $false `
-VM $workers `
-Enabled $true
# verify placement is actually spread
Get-VM ocp-worker-* | Select Name, VMHost | Sort VMHost06. Build layer 4: OpenShift and the CSI driver
# storageclass pointing at the VMstore-backed datastore
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: tintri-nfs-prod
provisioner: csi.tintri.com
parameters:
datastore: "OCP-PROD-01"
zone: "prod-a"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumerTwo parameters that matter more than they look.
reclaimPolicy: Retainmeans deleting a PVC does not delete the data, which you want for a database.volumeBindingMode: WaitForFirstConsumerdelays provisioning until the pod is scheduled, so the volume is created where the pod actually lands rather than somewhere the scheduler later rejects.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: postgres
replicas: 3
selector:
matchLabels: { app: postgres }
template:
metadata:
labels: { app: postgres }
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels: { app: postgres }
topologyKey: kubernetes.io/hostname
containers:
- name: postgres
image: registry.redhat.io/rhel9/postgresql-15
volumeMounts:
- name: data
mountPath: /var/lib/pgsql/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: tintri-nfs-prod
resources:
requests:
storage: 200GiFor the shared artifact volume, a separate RWX claim. RWX is where NFS earns its place, because the same volume mounts read-write on several pods at once without a clustered filesystem.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: build-artifacts
spec:
accessModes: ["ReadWriteMany"]
storageClassName: tintri-nfs-prod
resources:
requests:
storage: 500Gi07. Validate before you hand it over
| Test | Command | Pass condition |
|---|---|---|
| Jumbo path | vmkping -I vmk1 -s 8972 -d <array> | Zero loss from every host |
| Datastore visible | esxcli storage nfs list | Accessible true on every host |
| Provisioning | kubectl get pvc | Bound within seconds |
| RWX concurrency | Two pods write the same volume | Both succeed, no lock errors |
| Host failure | Power off one ESXi host | Workers restart, at most one replica lost |
| Fabric failure | Reboot fabric interconnect B | No datastore APD, no pod restarts |
| Baseline | fio in a pod, 30 minutes | Record it, you will need it later |
That last row is the one people skip and then regret. Without a recorded baseline, every future performance complaint is unfalsifiable.
08. Triage: which layer owns the latency
Before opening any tool, collect three numbers for the same time window: what the guest sees, what ESXi sees, and what the array reports. The pattern tells you where to go.
GUEST HIGH | ESXi HIGH | ARRAY HIGH -> the array is genuinely saturated
GUEST HIGH | ESXi HIGH | ARRAY LOW -> network or host queue. UCS + esxtop
GUEST HIGH | ESXi LOW | ARRAY LOW -> the guest. CPU ready, co-stop, fs
GUEST LOW | ESXi HIGH | ARRAY LOW -> measurement window mismatch, recheck
ALL LOW, app slow -> not storage. Look at CPU and memoryThe second row is the expensive one. Guest and host high, array low, is the signature of an MTU mismatch, a saturated uplink or a queue depth problem. It is also the case most often misfiled as a storage ticket, because the symptom is slow disk and the array is the obvious suspect.
09. Runbook: esxtop
SSH to the host, run esxtop, then switch views with single keys.
| Key | View | Use it for |
|---|---|---|
c | CPU | Ready time, co-stop, limits |
m | Memory | Ballooning, swap, NUMA locality |
n | Network | Dropped packets per vNIC and vmnic |
d | Disk adapter | Block adapters. Mostly empty on pure NFS |
u | Disk device | Block devices. Not useful for NFS datastores |
v | Disk VM | This is the NFS view. Per-VM virtual disk latency |
The NFS trap. Most esxtop guides tell you to look at DAVG in the
uview. On an NFS datastore there is no SCSI device, so those counters stay empty and people conclude the storage is idle while the database is crawling. For NFS you use thevview and readLAT/rdandLAT/wr. Learn this once and it saves you every time.
Counters and thresholds
| Counter | View | Meaning | Investigate above |
|---|---|---|---|
%RDY | c | vCPU ready to run, waiting for a core | 10 percent per vCPU |
%CSTP | c | Co-stop. Wide VM being held back | 3 percent |
%MLMTD | c | Held back by a configured CPU limit | Any non-zero value |
MCTLSZ | m | Balloon size. Host reclaiming memory | Any non-zero value |
SWCUR, SWR/s | m | Host swapping guest memory | Any non-zero value, act now |
N%L | m | NUMA locality | Below 80 percent |
%DRPTX, %DRPRX | n | Dropped packets | Any non-zero value |
LAT/rd, LAT/wr | v | Virtual disk latency in ms, works on NFS | 20 ms sustained |
DAVG/cmd | u | Device latency, array plus fabric | 20 ms, block only |
KAVG/cmd | u | Kernel latency, queueing | 2 ms, block only |
QUED | u | Commands queued | Persistently non-zero |
Interactive esxtop is for the moment you are in front of it. For an incident that happens overnight, capture in batch mode.
# all counters, every 10 seconds, for one hour
esxtop -b -a -d 10 -n 360 > /tmp/esxtop-$(hostname)-$(date +%Y%m%d-%H%M).csv
# compress before moving it, these get large fast
gzip /tmp/esxtop-*.csv
# to limit the field set interactively first, then save a config:
# run esxtop, press the view key, use f to toggle fields, then W to write ~/.esxtop50rcWhen esxtop shows high latency but you need the distribution rather than the average, use vscsiStats. Averages hide the tail, and the tail is what the application feels.
# list worlds and find the VM
vscsiStats -l
# start collection for one VM
vscsiStats -s -w <worldGroupID>
# latency histogram
vscsiStats -p latency -w <worldGroupID>
# io size histogram, tells you if the guest is doing what you think
vscsiStats -p ioLength -w <worldGroupID>
# always stop it, it does not expire cleanly on its own
vscsiStats -x -w <worldGroupID>10. Runbook: NFS and VMstore
State and reachability
# is it mounted and accessible on this host
esxcli storage nfs list
# NFS 4.1 mounts are listed separately, check both
esxcli storage nfs41 list
# which vmkernel is actually being used for the storage subnet
esxcli network ip route ipv4 list
esxcli network ip interface ipv4 get
# re-run the jumbo test, it is the fastest way to catch a fabric change
vmkping -I vmk1 -s 8972 -d 10.20.30.10
# capture live traffic if you need proof
pktcap-uw --vmk vmk1 --dir 2 -o /tmp/vmk1.pcapWhat to grep for in the logs
grep -iE 'nfs|apd|pdl' /var/log/vmkernel.log | tail -100
# specific strings worth knowing by heart
grep -i "Lost connection to server" /var/log/vmkernel.log
grep -i "has entered the all paths down" /var/log/vmkernel.log
grep -i "has exited the all paths down" /var/log/vmkernel.log
grep -i "NFSLock" /var/log/vmkernel.log
grep -i "StorageApdHandler" /var/log/vmkernel.log| Symptom | Most likely cause | First check |
|---|---|---|
| Mounts, then stalls under load | MTU mismatch on one hop | vmkping -s 8972 -d from every host |
| Accessible false on one host only | Export restriction or VLAN on that blade | Export list, then that blade’s vNIC config |
| Repeated APD entered and exited | Flapping uplink or heartbeat timeout | Switch interface counters, then heartbeat settings |
| Mount fails, others fine | MaxVolumes or TCP/IP heap exhausted | NFS.MaxVolumes and Net.TcpipHeapMax |
| VM will not power on, lock error | Stale lock from a crashed host | Identify lock owner MAC in vmkernel.log |
| Array latency low, guest latency high | Not the array | Go to the switch runbook |
On the VMstore side
- Read the per-VM latency breakdown. The array separates host, network, storage and contention components, which is exactly the split you need for the triage table above.
- Confirm the Hypervisor Manager registration is still healthy. When it lapses, per-VM data quietly degrades to file-level data and your correlation breaks.
- Check whether a QoS floor or ceiling is applied to the VM you are investigating. A ceiling someone set during a past incident and never removed is a common cause of unexplained throttling.
- Check snapshot count on the affected VMs. Long snapshot chains change the write path and the performance profile with it.
11. Runbook: UCS fabric and upstream switches
This is the layer that produces storage symptoms while reporting network health. Work it in order: MTU, then errors, then saturation, then topology.
On the fabric interconnect
connect nxos a
# MTU, the first thing to rule out
show interface ethernet 1/1 | include MTU
show queuing interface ethernet 1/1 | include MTU
# errors: CRC, input discards, runts, giants
show interface ethernet 1/1 counters errors
show interface counters errors | exclude 0
# saturation on the uplinks
show interface ethernet 1/1 | include rate
show port-channel summary
show port-channel traffic
# pause frames. non-zero here explains intermittent latency
show interface priority-flow-control
show interface ethernet 1/1 | include Pause
# what is on the other end
show cdp neighbors
show lldp neighborsOn the upstream Nexus
# the storage VLAN must be allowed on every trunk in the path
show interface trunk
show vlan brief | include <storage-vlan>
# system level jumbo policy
show policy-map system type network-qos
show running-config | include jumbo
# error counters, clear then re-check under load
show interface ethernet 1/5 counters errors
clear counters interface ethernet 1/5
# is the array MAC where you expect it
show mac address-table vlan <storage-vlan>
# vPC consistency, a mismatch here breaks one path only
show vpc
show vpc consistency-parameters global| Finding | What it means | Action |
|---|---|---|
| Input discards climbing | Buffer exhaustion, usually a speed mismatch or a burst | Check uplink utilisation and QoS buffering |
| CRC errors | Physical layer: cable, optic or port | Replace the optic, then the cable, then move ports |
| Pause frames non-zero | Flow control throttling the link | Correlate timestamps with the latency spikes |
| MTU differs on one hop | The classic silent jumbo failure | Fix and re-run the 8972 vmkping |
| Storage VLAN missing on a trunk | One path is dead, teaming hides it | Add the VLAN, then test failover deliberately |
| vPC consistency mismatch | Half your redundancy is not there | Reconcile before the next maintenance window |
12. Runbook: Kubernetes and CSI
# the claim and what the provisioner did with it
kubectl get pvc -A
kubectl describe pvc data-postgres-0
kubectl get pv
# attachment state, this is where stuck volumes show up
kubectl get volumeattachment
kubectl describe volumeattachment <name>
# pod level events. FailedMount and its exact message matter
kubectl describe pod postgres-0 | sed -n '/Events/,$p'
# the driver itself, one container per responsibility
kubectl -n tintri-csi get pods
kubectl -n tintri-csi logs <controller-pod> -c csi-provisioner --tail=200
kubectl -n tintri-csi logs <controller-pod> -c csi-attacher --tail=200
kubectl -n tintri-csi logs <node-pod> -c csi-node-driver --tail=200
# from inside the worker VM, what the kernel actually mounted
mount | grep nfs
nfsstat -c| Symptom | Layer that actually owns it | Next step |
|---|---|---|
| PVC stuck Pending | CSI provisioner or StorageClass | Provisioner logs, then datastore free space |
| FailedMount, timeout | Network or export, not Kubernetes | vmkping from the host, then export list |
| Volume attaches, pod slow | ESXi, fabric or array | Start at the triage table |
| RWX mounts on one pod only | Access mode or export permission | Confirm RWX on the PV, then the export |
| Stuck terminating after node loss | Stale VolumeAttachment | Confirm the node is gone, then clear it |
| Everything healthy, app slow | Probably CPU, not storage | esxtop c view, %RDY and %CSTP |
Closing thought
The build in the first half of this post takes a day. The runbook in the second half is what determines whether the platform is supportable for the next three years. If you take one thing from it, take the triage table in section 08: collect guest, host and array latency for the same window before you open any tool. That single habit turns a four-team bridge call into a five-minute answer.
Commands here were used against vSphere 8.0, UCS Manager 4.x, NX-OS 9.x and OpenShift 4.18. Counter names and esxcli namespaces are stable across recent versions, but verify advanced settings against current vendor guidance rather than copying values, including from this post. Nothing here is official guidance from any vendor.
Leave a Reply