Automating the jumbo frame check: vmkping across every ESXi host

Part 5 of 10. The datastore mounts, the dashboard is green, and then the database goes slow at 2am. A partial jumbo configuration passes every check except the one that matters, which is sending a full sized frame that must not be fragmented.

The arithmetic, and why 8972

9000   MTU on the vmkernel and the array data interface
  -20  IP header
  -8   ICMP header
 ----
 8972  the largest payload that fits without fragmenting

The -d flag sets do-not-fragment. Without it the test always
passes, because the stack quietly splits the packet for you and
you learn nothing.

Always test both sizes. Small passes and large fails means an MTU mismatch on one hop. Both fail means VLAN, routing or an export permission. Both pass means the path is genuinely clean. One test on its own cannot tell those apart.

PowerShell

PowerCLI exposes the ping diagnostic through esxcli, so this runs from your workstation against every host with no SSH enabled anywhere.

#requires -Modules VMware.PowerCLI
<#
  jumbo-sweep.ps1
  Checks vmkernel MTU config, then proves the path with a
  do-not-fragment ping at full frame size from every host.
#>

param(
    [Parameter(Mandatory)][string]$VCenter,
    [Parameter(Mandatory)][pscredential]$Credential,
    [Parameter(Mandatory)][string]$StorageTarget,   # VMstore data IP
    [string]$VmkPattern = 'vmk1',
    [int]$Mtu = 9000
)

$payload = $Mtu - 28
Connect-VIServer -Server $VCenter -Credential $Credential | Out-Null

try {
    $results = foreach ($h in (Get-VMHost | Where-Object ConnectionState -eq 'Connected')) {

        $vmk = Get-VMHostNetworkAdapter -VMHost $h -VMKernel |
               Where-Object { $_.Name -eq $VmkPattern }

        if (-not $vmk) {
            [pscustomobject]@{ Host=$h.Name; VMK='(absent)'; ConfigMtu=$null
                               Small='n/a'; Jumbo='n/a'; Verdict='NO VMKERNEL' }
            continue
        }

        $esxcli = Get-EsxCli -VMHost $h -V2

        function Test-Ping([int]$size) {
            $args = $esxcli.network.diag.ping.CreateArgs()
            $args.host      = $StorageTarget
            $args.interface = $VmkPattern
            $args.size      = $size
            $args.df        = $true
            $args.count     = 3
            try {
                $r = $esxcli.network.diag.ping.Invoke($args)
                return ([int]$r.Summary.Recieved -gt 0)
            } catch { return $false }
        }

        $small = Test-Ping 1472
        $jumbo = Test-Ping $payload

        $verdict = if     ($small -and $jumbo) { 'ok' }
                   elseif ($small -and -not $jumbo) { 'MTU MISMATCH' }
                   elseif (-not $small) { 'NO PATH' }
                   else { 'ODD' }

        [pscustomobject]@{
            Host      = $h.Name
            VMK       = $vmk.Name
            ConfigMtu = $vmk.Mtu
            Small     = $small
            Jumbo     = $jumbo
            Verdict   = $verdict
        }
    }

    $results | Format-Table -AutoSize

    $fail = $results | Where-Object Verdict -ne 'ok'
    if ($fail) {
        Write-Host "`n$($fail.Count) host(s) failed. Do not mount or migrate onto this path." `
                   -ForegroundColor Red
        exit 2
    }
    Write-Host "`nAll hosts clean at $Mtu MTU." -ForegroundColor Green
}
finally {
    Disconnect-VIServer -Confirm:$false -ErrorAction SilentlyContinue
}

Python

The same esxcli namespace is reachable from pyVmomi, though the call is less friendly. This version also reports the configured MTU on the vmkernel and its virtual switch, which catches the case where the vmk says 9000 but the uplink does not.

#!/usr/bin/env python3
"""jumbo_sweep.py - verify configured MTU and prove the path with vmkping.

Config checking uses pyVmomi. The live ping uses pyvmomi's esxcli passthrough
where available; if your build does not expose it, fall back to running
  vmkping -I vmk1 -s 8972 -d <target>
over SSH and feed the exit code into this report.
"""

import argparse
import ssl
import sys

from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim


def mtu_report(si, vmk_name, expected):
    content = si.RetrieveContent()
    view = content.viewManager.CreateContainerView(
        content.rootFolder, [vim.HostSystem], True)
    hosts = list(view.view)
    view.Destroy()

    rows = []
    for h in hosts:
        if h.runtime.connectionState != "connected":
            continue

        net = h.config.network
        vnic = next((v for v in net.vnic if v.device == vmk_name), None)

        # switch MTU: standard switches expose it directly
        sw_mtu = None
        if vnic and vnic.portgroup:
            pg = next((p for p in net.portgroup
                       if p.spec.name == vnic.portgroup), None)
            if pg:
                vs = next((s for s in net.vswitch
                           if s.name == pg.spec.vswitchName), None)
                if vs:
                    sw_mtu = vs.mtu

        rows.append({
            "host": h.name,
            "vmk": vmk_name if vnic else "(absent)",
            "ip": vnic.spec.ip.ipAddress if vnic else "-",
            "vmk_mtu": vnic.spec.mtu if vnic else None,
            "switch_mtu": sw_mtu,
            "ok": bool(vnic) and vnic.spec.mtu == expected
                  and (sw_mtu is None or sw_mtu >= expected),
        })
    return rows


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--vcenter", required=True)
    p.add_argument("--user", required=True)
    p.add_argument("--password", required=True)
    p.add_argument("--vmk", default="vmk1")
    p.add_argument("--mtu", type=int, default=9000)
    a = p.parse_args()

    ctx = ssl._create_unverified_context()
    si = SmartConnect(host=a.vcenter, user=a.user, pwd=a.password,
                      sslContext=ctx)
    try:
        rows = mtu_report(si, a.vmk, a.mtu)
    finally:
        Disconnect(si)

    print(f"Expecting MTU {a.mtu} on {a.vmk}")
    print(f"Payload for a do-not-fragment ping: {a.mtu - 28}\n")

    hdr = f"{'HOST':<30}{'VMK':<8}{'IP':<16}{'VMK MTU':>9}{'SW MTU':>8}   RESULT"
    print(hdr)
    print("-" * (len(hdr) + 4))

    bad = 0
    for r in rows:
        if not r["ok"]:
            bad += 1
        print(f"{r['host']:<30}{r['vmk']:<8}{str(r['ip']):<16}"
              f"{str(r['vmk_mtu']):>9}{str(r['switch_mtu']):>8}"
              f"   {'ok' if r['ok'] else 'MISCONFIGURED'}")

    print(f"\nHosts with MTU config problems: {bad}")
    print("\nConfig being right does not prove the path. Run this on each host:")
    print(f"  vmkping -I {a.vmk} -s {a.mtu - 28} -d <array-data-ip>")
    print(f"  vmkping -I {a.vmk} -s 1472 -d <array-data-ip>")
    return 0 if bad == 0 else 2


if __name__ == "__main__":
    sys.exit(main())

Where the mismatch usually is

Hop	Setting	Check
vmkernel	9000	esxcli network ip interface list
vSwitch or DVS uplink	9000	The script above reports it
UCS vNIC template	9000	UCS Manager, LAN, vNIC template
UCS QoS system class	9216	show queuing interface ethernet 1/1
Upstream Nexus	9216 policy	show policy-map system type network-qos
Array data interface	9000	VMstore network settings

The fabric numbers are 9216, not 9000, because they include headers. A vNIC set to 9000 bound to a QoS class still at 1500 is the single most common silent failure in this whole stack.

Next: snapshot sprawl on the VMstore, and how much capacity it is quietly holding.

The esxcli ping namespace and its result properties vary between vSphere versions. Confirm the returned object on your build, particularly the spelling of the received count, before relying on the verdict logic.