NFS datastore health sweep across a VCF fleet: finding the one host that mounted differently

Part 4 of 10. NFS datastore problems are rarely fleet wide. They are almost always one host that mounted against the wrong address, missed the mount entirely, or reports inaccessible while the other eleven look fine. This sweep finds it in seconds.

What drift actually looks like

Drift	Symptom	Cause
Datastore missing on one host	VM will not power on or vMotion there	Host added after the mount, never remounted
Same datastore, different server IP	Works, until you fail over a path	Mounted by hostname on some, IP on others
Accessible false	APD warnings, intermittent stalls	Export permission, VLAN or MTU on that blade
Different datastore name, same export	vMotion fails with a datastore mismatch	Typo during a manual mount
Read only on one host	Cannot write, reads fine, very confusing	Export ACL for that host is read only

Mount by IP or by FQDN, but pick one and enforce it. ESXi treats 10.20.30.10:/export and vmstore.lab.local:/export as two different datastores even though they are the same storage. That mismatch is invisible in the vSphere Client until a migration fails.

PowerShell

#requires -Modules VMware.PowerCLI
<#
  nfs-health-sweep.ps1
  Compares NFS mounts across every host and reports the outliers.
#>

param(
    [Parameter(Mandatory)][string]$VCenter,
    [Parameter(Mandatory)][pscredential]$Credential,
    [string]$Cluster = '*'
)

Connect-VIServer -Server $VCenter -Credential $Credential | Out-Null

try {
    $hosts = Get-VMHost -Location (Get-Cluster -Name $Cluster) |
             Where-Object { $_.ConnectionState -eq 'Connected' }

    $mounts = foreach ($h in $hosts) {
        $esxcli = Get-EsxCli -VMHost $h -V2
        foreach ($n in $esxcli.storage.nfs.list.Invoke()) {
            [pscustomobject]@{
                Host       = $h.Name
                Datastore  = $n.VolumeName
                Server     = $n.Host
                Share      = $n.Share
                Accessible = $n.Accessible
                ReadOnly   = $n.ReadOnly
                Key        = "$($n.Host):$($n.Share)"
            }
        }
        # NFS 4.1 mounts live in a separate namespace, check both
        try {
            foreach ($n in $esxcli.storage.nfs41.list.Invoke()) {
                [pscustomobject]@{
                    Host       = $h.Name
                    Datastore  = $n.VolumeName
                    Server     = ($n.Host -join ',')
                    Share      = $n.Share
                    Accessible = $n.Accessible
                    ReadOnly   = $n.ReadOnly
                    Key        = "$($n.Host -join ','):$($n.Share)"
                }
            }
        } catch { }
    }

    $hostCount = $hosts.Count
    Write-Host "Hosts checked: $hostCount`n" -ForegroundColor Cyan

    # 1. datastores not present everywhere
    Write-Host 'MOUNT COVERAGE' -ForegroundColor Cyan
    $mounts | Group-Object Datastore | ForEach-Object {
        $n = ($_.Group.Host | Select-Object -Unique).Count
        $flag = if ($n -eq $hostCount) { 'ok' } else { 'MISSING' }
        "  {0,-30} {1,2}/{2} hosts  [{3}]" -f $_.Name, $n, $hostCount, $flag

        if ($n -ne $hostCount) {
            $have = $_.Group.Host | Select-Object -Unique
            foreach ($m in ($hosts.Name | Where-Object { $_ -notin $have })) {
                "      not mounted on: $m"
            }
        }
    }

    # 2. same datastore name, different server or share
    Write-Host "`nADDRESS CONSISTENCY" -ForegroundColor Cyan
    $mounts | Group-Object Datastore | ForEach-Object {
        $keys = $_.Group.Key | Select-Object -Unique
        if ($keys.Count -gt 1) {
            Write-Host "  MISMATCH on $($_.Name)" -ForegroundColor Red
            foreach ($k in $keys) {
                $who = ($_.Group | Where-Object Key -eq $k).Host -join ', '
                "      $k  ->  $who"
            }
        }
    }

    # 3. accessible / read only problems
    Write-Host "`nSTATE" -ForegroundColor Cyan
    $bad = $mounts | Where-Object { -not $_.Accessible -or $_.ReadOnly }
    if ($bad) {
        $bad | Format-Table Host, Datastore, Accessible, ReadOnly -AutoSize
    } else {
        "  all mounts accessible and read-write"
    }
}
finally {
    Disconnect-VIServer -Confirm:$false -ErrorAction SilentlyContinue
}

Python

pyVmomi reads the mounted filesystem list straight from each host’s config, no SSH required.

#!/usr/bin/env python3
"""nfs_health_sweep.py - find NFS mount drift across a vSphere cluster."""

import argparse
import ssl
import sys
from collections import defaultdict

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim


def collect(si):
    content = si.RetrieveContent()
    view = content.viewManager.CreateContainerView(
        content.rootFolder, [vim.HostSystem], True)
    hosts = list(view.view)
    view.Destroy()

    mounts = []
    live = []
    for h in hosts:
        if h.runtime.connectionState != "connected":
            continue
        live.append(h.name)
        for mnt in h.config.fileSystemVolume.mountInfo:
            vol = mnt.volume
            if not isinstance(vol, vim.host.NasVolume):
                continue
            mounts.append({
                "host": h.name,
                "datastore": vol.name,
                "server": vol.remoteHost,
                "share": vol.remotePath,
                "type": vol.type,
                "accessible": bool(mnt.mountInfo.accessible),
                "mode": mnt.mountInfo.accessMode,
                "key": f"{vol.remoteHost}:{vol.remotePath}",
            })
    return live, mounts


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--vcenter", required=True)
    p.add_argument("--user", required=True)
    p.add_argument("--password", required=True)
    a = p.parse_args()

    ctx = ssl._create_unverified_context()
    si = SmartConnect(host=a.vcenter, user=a.user, pwd=a.password,
                      sslContext=ctx)
    try:
        hosts, mounts = collect(si)
    finally:
        Disconnect(si)

    total = len(hosts)
    print(f"Hosts checked: {total}\n")

    by_ds = defaultdict(list)
    for m in mounts:
        by_ds[m["datastore"]].append(m)

    issues = 0

    print("MOUNT COVERAGE")
    for ds, rows in sorted(by_ds.items()):
        have = {r["host"] for r in rows}
        flag = "ok" if len(have) == total else "MISSING"
        if flag != "ok":
            issues += 1
        print(f"  {ds:<30} {len(have):>2}/{total} hosts  [{flag}]")
        for miss in sorted(set(hosts) - have):
            print(f"      not mounted on: {miss}")

    print("\nADDRESS CONSISTENCY")
    clean = True
    for ds, rows in sorted(by_ds.items()):
        keys = defaultdict(list)
        for r in rows:
            keys[r["key"]].append(r["host"])
        if len(keys) > 1:
            clean = False
            issues += 1
            print(f"  MISMATCH on {ds}")
            for k, who in keys.items():
                print(f"      {k}  ->  {', '.join(sorted(who))}")
    if clean:
        print("  every datastore mounted from one address")

    print("\nSTATE")
    bad = [m for m in mounts
           if not m["accessible"] or m["mode"] != "readWrite"]
    if bad:
        issues += len(bad)
        for m in bad:
            print(f"  {m['host']:<28} {m['datastore']:<24} "
                  f"accessible={m['accessible']} mode={m['mode']}")
    else:
        print("  all mounts accessible and read-write")

    print(f"\nIssues: {issues}")
    return 0 if issues == 0 else 2


if __name__ == "__main__":
    sys.exit(main())

Run it after every host addition and after any maintenance window. Drift is introduced by change, not by time.

Next: proving jumbo frames actually work on every host, because a clean mount on a broken MTU path is the failure that waits for load.

Read only vCenter access is sufficient. The esxcli namespace differs slightly between vSphere versions, so confirm the properties returned on your build before scheduling this.