Auditing Tintri per-VM QoS: finding the ceiling nobody removed

Part 7 of 10. Someone caps a noisy VM during an incident at 2am. The incident closes. The cap stays. Eighteen months later a different engineer is investigating why that VM will not go faster, and nothing in vSphere explains it.

Floors and ceilings do different damage

Setting	Purpose	Failure mode when stale
Floor (min IOPS)	Guarantee a VM gets at least this much	Reserves capacity for something decommissioned, starving everything else
Ceiling (max IOPS)	Stop a VM consuming everything	Silently throttles a VM forever, invisible from vSphere

A ceiling shows up as contention latency, not as an error. Cross reference this audit with part 2. If a VM’s dominant latency component is contention and it has a ceiling set, you have found your answer without touching the fabric or the hypervisor.

PowerShell

#requires -Version 5.1
<#
  tintri-qos-audit.ps1
  Lists every VM with a QoS floor or ceiling and flags the ones
  actually being throttled right now.
#>

param(
    [Parameter(Mandatory)][string]$VMstore,
    [Parameter(Mandatory)][pscredential]$Credential,
    [switch]$OnlyConfigured,
    [string]$CsvPath
)

Import-Module TintriPSToolkit -ErrorAction Stop
$conn = Connect-TintriServer -Server $VMstore -Credential $Credential -SetDefaultServer

try {
    $rows = foreach ($vm in Get-TintriVM) {

        $qos  = $vm.QosConfig
        $min  = [int]$qos.MinNormalizedIops
        $max  = [int]$qos.MaxNormalizedIops

        if ($OnlyConfigured -and $min -eq 0 -and $max -eq 0) { continue }

        $stat = $vm.Stat.SortedStats | Select-Object -Last 1
        $iops = if ($stat) { [math]::Round($stat.OperationsTotalIops, 0) } else { 0 }
        $cont = if ($stat) { [math]::Round($stat.LatencyContentionMs, 2) } else { 0 }

        # a ceiling is biting if we are close to it and contention is present
        $throttled = ($max -gt 0) -and ($iops -ge ($max * 0.9)) -and ($cont -gt 0.5)

        [pscustomobject]@{
            VM          = $vm.VmwareVM.Name
            FloorIops   = if ($min -gt 0) { $min } else { '-' }
            CeilingIops = if ($max -gt 0) { $max } else { '-' }
            CurrentIops = $iops
            ContentionMs= $cont
            Status      = if     ($throttled)        { 'THROTTLED NOW' }
                          elseif ($max -gt 0)        { 'ceiling set' }
                          elseif ($min -gt 0)        { 'floor set' }
                          else                       { 'unset' }
        }
    }

    $rows | Sort-Object Status, VM | Format-Table -AutoSize

    $ceil  = @($rows | Where-Object { $_.CeilingIops -ne '-' })
    $floor = @($rows | Where-Object { $_.FloorIops   -ne '-' })
    $hot   = @($rows | Where-Object Status -eq 'THROTTLED NOW')

    Write-Host "`nSUMMARY" -ForegroundColor Cyan
    "  VMs with a ceiling : {0}" -f $ceil.Count
    "  VMs with a floor   : {0}" -f $floor.Count
    "  Throttled right now: {0}" -f $hot.Count

    if ($hot) {
        Write-Host "`nACTIVELY THROTTLED" -ForegroundColor Red
        $hot | ForEach-Object {
            "  {0,-36} {1} IOPS against a ceiling of {2}, contention {3} ms" -f `
                $_.VM, $_.CurrentIops, $_.CeilingIops, $_.ContentionMs
        }
        Write-Host "`nConfirm each of these is still intentional." -ForegroundColor Yellow
    }

    if ($CsvPath) {
        $rows | Export-Csv -Path $CsvPath -NoTypeInformation
        Write-Host "`nWritten to $CsvPath" -ForegroundColor Green
    }
}
finally {
    if ($conn) { Disconnect-TintriServer -TintriServer $conn }
}

Python

#!/usr/bin/env python3
"""tintri_qos_audit.py - report every per-VM QoS floor and ceiling.

Read only. Changing QoS is a deliberate act and is not automated here.
"""

import argparse
import os
import sys

import urllib3
import requests

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
CREDS_TYPE = "com.tintri.api.rest.vcommon.dto.rbac.RestApiCredentials"


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--vmstore", required=True)
    p.add_argument("--user", default=os.environ.get("TINTRI_USER"))
    p.add_argument("--password", default=os.environ.get("TINTRI_PASS"))
    p.add_argument("--api-version", default="v310")
    p.add_argument("--only-configured", action="store_true")
    a = p.parse_args()

    if not a.user or not a.password:
        sys.exit("set TINTRI_USER and TINTRI_PASS")

    base = f"https://{a.vmstore}/api/{a.api_version}"
    s = requests.Session()
    s.verify = False
    r = s.post(f"{base}/session/login",
               json={"typeId": CREDS_TYPE,
                     "username": a.user, "password": a.password},
               timeout=30)
    if r.status_code == 404:
        sys.exit(f"404 on {base}. Try v310.203 or check GET /api/info.")
    r.raise_for_status()

    rows = []
    try:
        vms = s.get(f"{base}/vm", timeout=60).json().get("items", [])
        for vm in vms:
            name = (vm.get("vmware") or {}).get("name", "unknown")
            qos = vm.get("qosConfig") or {}
            floor = int(qos.get("minNormalizedIops") or 0)
            ceiling = int(qos.get("maxNormalizedIops") or 0)

            if a.only_configured and not floor and not ceiling:
                continue

            stats = (vm.get("stat") or {}).get("sortedStats") or []
            st = stats[-1] if stats else {}
            iops = float(st.get("operationsTotalIops") or 0)
            cont = float(st.get("latencyContentionMs") or 0)

            throttled = bool(ceiling) and iops >= ceiling * 0.9 and cont > 0.5
            status = ("THROTTLED NOW" if throttled
                      else "ceiling set" if ceiling
                      else "floor set" if floor
                      else "unset")

            rows.append({"vm": name, "floor": floor, "ceiling": ceiling,
                         "iops": iops, "contention": cont, "status": status})
    finally:
        s.get(f"{base}/session/logout", timeout=15)

    rows.sort(key=lambda r: (r["status"] != "THROTTLED NOW", r["vm"]))

    hdr = (f"{'VM':<36}{'FLOOR':>8}{'CEILING':>9}{'IOPS':>8}"
           f"{'CONT ms':>9}   STATUS")
    print(hdr)
    print("-" * (len(hdr) + 4))
    for r in rows:
        print(f"{r['vm'][:35]:<36}"
              f"{(r['floor'] or '-'):>8}"
              f"{(r['ceiling'] or '-'):>9}"
              f"{r['iops']:>8.0f}{r['contention']:>9.2f}   {r['status']}")

    hot = [r for r in rows if r["status"] == "THROTTLED NOW"]
    print(f"\nCeilings set : {sum(1 for r in rows if r['ceiling'])}")
    print(f"Floors set   : {sum(1 for r in rows if r['floor'])}")
    print(f"Throttled now: {len(hot)}")

    if hot:
        print("\nConfirm each of these is still intentional:")
        for r in hot:
            print(f"  {r['vm'][:35]:<36} {r['iops']:.0f} IOPS against "
                  f"a ceiling of {r['ceiling']}, contention {r['contention']:.2f} ms")

    return 0 if not hot else 2


if __name__ == "__main__":
    sys.exit(main())

Using it well

• Run it quarterly, not weekly. QoS changes rarely. A quarterly review with an owner named against every ceiling is the right cadence.
• Record why, not just what. The array stores the number, never the reason. Keep a change record next to each ceiling or the next engineer faces the same mystery.
• Do not automate the removal. Both scripts are read only on purpose. A ceiling that looks stale may be the only thing protecting a shared array from one badly behaved workload.

Next: certificate expiry across SDDC Manager, vCenter and NSX, which is the outage everyone schedules for themselves and then forgets about.

The QoS field names, particularly minNormalizedIops and maxNormalizedIops, vary by TxOS build. Dump one VM object and confirm before scheduling. The 0.9 and 0.5 thresholds in the throttling check are heuristics, tune them to your baseline.