Part 3 of 10. Parts 1 and 2 gave you two halves of an answer. This is the join: one table, one time window, three numbers per VM, and a verdict.
The logic
Pull what the hypervisor thinks the VM’s disk latency is, pull what the array says it delivered for the same VM, subtract, and the remainder is time spent between them. Everything else follows from the sign and size of that remainder.
GUEST what the application waited for
minus
ARRAY what the VMstore reports it took
equals
GAP hypervisor queue + fabric + everything else
GAP small, ARRAY high -> array is genuinely busy
GAP large, ARRAY low -> host queue or network. This is the common one.
BOTH low, app still slow -> not storage. Check CPU ready and co-stop.Same window or the comparison is meaningless. vSphere real time samples are 20 seconds. VMstore statistics are collected on their own cadence. If you compare a vSphere five minute rollup against a live array reading you will produce nonsense and act on it. Both scripts below take an explicit window and state it in the output.
PowerShell
PowerCLI for the vSphere side, the Tintri toolkit for the array side, matched on VM name.
#requires -Modules VMware.PowerCLI
<#
latency-triage.ps1
Joins vSphere virtual disk latency to Tintri per-VM latency and
names the layer that owns the delay.
#>
param(
[Parameter(Mandatory)][string]$VCenter,
[Parameter(Mandatory)][string]$VMstore,
[Parameter(Mandatory)][pscredential]$VcCredential,
[Parameter(Mandatory)][pscredential]$TintriCredential,
[int]$WindowMinutes = 15,
[double]$GuestAlertMs = 20
)
Import-Module TintriPSToolkit -ErrorAction Stop
$start = (Get-Date).AddMinutes(-$WindowMinutes)
$end = Get-Date
Write-Host ("Window: {0:HH:mm:ss} to {1:HH:mm:ss}" -f $start, $end) `
-ForegroundColor Cyan
Connect-VIServer -Server $VCenter -Credential $VcCredential | Out-Null
$tintri = Connect-TintriServer -Server $VMstore `
-Credential $TintriCredential -SetDefaultServer
try {
# ---- vSphere side ----
# totalLatency covers read and write at the virtual disk layer
$vsphere = @{}
$vms = Get-VM | Where-Object { $_.PowerState -eq 'PoweredOn' }
$stats = Get-Stat -Entity $vms `
-Stat 'virtualDisk.totalWriteLatency.average',
'virtualDisk.totalReadLatency.average' `
-Start $start -Finish $end -ErrorAction SilentlyContinue
foreach ($g in ($stats | Group-Object { $_.Entity.Name })) {
$vsphere[$g.Name] = [math]::Round(
($g.Group | Measure-Object Value -Average).Average, 2)
}
# ---- Tintri side ----
$array = @{}
foreach ($tvm in Get-TintriVM) {
$s = $tvm.Stat.SortedStats | Select-Object -Last 1
if ($s) {
$array[$tvm.VmwareVM.Name] = [pscustomobject]@{
Total = [math]::Round($s.LatencyTotalMs, 2)
Storage = [math]::Round($s.LatencyStorageMs, 2)
Network = [math]::Round($s.LatencyNetworkMs, 2)
Contention = [math]::Round($s.LatencyContentionMs, 2)
}
}
}
# ---- join and verdict ----
$report = foreach ($name in $vsphere.Keys) {
if (-not $array.ContainsKey($name)) { continue }
$guest = $vsphere[$name]
$a = $array[$name]
$gap = [math]::Round($guest - $a.Storage, 2)
$verdict = switch ($true) {
($guest -lt $GuestAlertMs) { 'ok'; break }
($a.Storage -ge ($guest * 0.7)) { 'ARRAY'; break }
($a.Network -gt 1) { 'NETWORK'; break }
($a.Contention -gt 1) { 'CONTENTION or QoS'; break }
default { 'HOST or FABRIC' }
}
[pscustomobject]@{
VM = $name
GuestMs = $guest
ArrayMs = $a.Storage
GapMs = $gap
NetMs = $a.Network
ContMs = $a.Contention
Verdict = $verdict
}
}
$report |
Sort-Object GuestMs -Descending |
Format-Table -AutoSize
$bad = $report | Where-Object { $_.Verdict -ne 'ok' }
Write-Host "`nVMs needing attention: $($bad.Count)" -ForegroundColor Yellow
}
finally {
if ($tintri) { Disconnect-TintriServer -TintriServer $tintri }
Disconnect-VIServer -Confirm:$false -ErrorAction SilentlyContinue
}Python
pyVmomi for vSphere performance counters, plus the VMstore session class from part 2. This is the version to schedule.
#!/usr/bin/env python3
"""latency_triage.py - join vSphere and Tintri latency, name the guilty layer."""
import argparse
import os
import ssl
import sys
from datetime import datetime, timedelta
import urllib3
import requests
from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
CREDS_TYPE = "com.tintri.api.rest.vcommon.dto.rbac.RestApiCredentials"
def vsphere_latency(host, user, pwd, minutes):
"""Average virtual disk latency per powered-on VM over the window."""
ctx = ssl._create_unverified_context()
si = SmartConnect(host=host, user=user, pwd=pwd, sslContext=ctx)
try:
content = si.RetrieveContent()
perf = content.perfManager
wanted = {"virtualDisk.totalReadLatency.average",
"virtualDisk.totalWriteLatency.average"}
ids = [c.key for c in perf.perfCounter
if f"{c.groupInfo.key}.{c.nameInfo.key}.{c.rollupType}" in wanted]
view = content.viewManager.CreateContainerView(
content.rootFolder, [vim.VirtualMachine], True)
vms = [v for v in view.view if v.runtime.powerState == "poweredOn"]
view.Destroy()
end = datetime.now()
start = end - timedelta(minutes=minutes)
out = {}
for vm in vms:
metrics = [vim.PerformanceManager.MetricId(counterId=i, instance="*")
for i in ids]
spec = vim.PerformanceManager.QuerySpec(
entity=vm, metricId=metrics,
startTime=start, endTime=end, intervalId=20)
try:
res = perf.QueryPerf(querySpec=[spec])
except Exception:
continue
vals = []
for r in res:
for s in r.value:
vals.extend([v for v in s.value if v >= 0])
if vals:
out[vm.name] = round(sum(vals) / len(vals), 2)
return out
finally:
Disconnect(si)
def tintri_latency(host, user, pwd, version="v310"):
base = f"https://{host}/api/{version}"
s = requests.Session()
s.verify = False
r = s.post(f"{base}/session/login",
json={"typeId": CREDS_TYPE, "username": user, "password": pwd},
timeout=30)
r.raise_for_status()
try:
data = s.get(f"{base}/vm", timeout=60).json()
out = {}
for vm in data.get("items", []):
name = (vm.get("vmware") or {}).get("name")
stats = (vm.get("stat") or {}).get("sortedStats") or []
if not name or not stats:
continue
st = stats[-1]
out[name] = {
"storage": float(st.get("latencyStorageMs") or 0),
"network": float(st.get("latencyNetworkMs") or 0),
"contention": float(st.get("latencyContentionMs") or 0),
}
return out
finally:
s.get(f"{base}/session/logout", timeout=15)
def verdict(guest, arr, threshold):
if guest < threshold:
return "ok"
if arr["storage"] >= guest * 0.7:
return "ARRAY"
if arr["network"] > 1:
return "NETWORK"
if arr["contention"] > 1:
return "CONTENTION or QoS"
return "HOST or FABRIC"
def main():
p = argparse.ArgumentParser()
p.add_argument("--vcenter", required=True)
p.add_argument("--vc-user", required=True)
p.add_argument("--vc-pass", default=os.environ.get("VC_PASS"))
p.add_argument("--vmstore", required=True)
p.add_argument("--tintri-user", required=True)
p.add_argument("--tintri-pass", default=os.environ.get("TINTRI_PASS"))
p.add_argument("--minutes", type=int, default=15)
p.add_argument("--alert-ms", type=float, default=20.0)
a = p.parse_args()
guest = vsphere_latency(a.vcenter, a.vc_user, a.vc_pass, a.minutes)
arr = tintri_latency(a.vmstore, a.tintri_user, a.tintri_pass)
print(f"Window: last {a.minutes} minutes, 20 second samples\n")
hdr = f"{'VM':<32}{'GUEST':>8}{'ARRAY':>8}{'GAP':>8}{'NET':>7}{'CONT':>7} VERDICT"
print(hdr)
print("-" * (len(hdr) + 6))
flagged = 0
for name in sorted(guest, key=guest.get, reverse=True):
if name not in arr:
continue
g = guest[name]
x = arr[name]
v = verdict(g, x, a.alert_ms)
if v != "ok":
flagged += 1
print(f"{name[:31]:<32}{g:>8.2f}{x['storage']:>8.2f}"
f"{g - x['storage']:>8.2f}{x['network']:>7.2f}"
f"{x['contention']:>7.2f} {v}")
print(f"\nFlagged: {flagged}")
return 0 if flagged == 0 else 2
if __name__ == "__main__":
sys.exit(main())Caveats worth stating out loud
- Matching on VM name is fragile. Duplicate names across folders or datacenters will mismatch. If that is a risk in your estate, join on instance UUID from both sides instead.
- The 0.7 ratio in the verdict is a heuristic, not physics. It says the array owns the delay if it accounts for most of it. Tune it once you have a baseline for your own environment.
- A HOST or FABRIC verdict is a starting point, not an answer. Take it to esxtop and to the switch counters. It has told you which door to knock on, nothing more.
Next in the series: sweeping NFS datastore health across every host in a VCF fleet, so you catch the one host that mounted differently.
Requires PowerCLI or pyVmomi plus network access to both vCenter and the VMstore. Read only accounts are sufficient on both sides. Verify counter and field names against your own build before scheduling this.
Leave a Reply