If you support more than three or four VMstores, the question you actually need answered each morning is not “how is this appliance”. It is “across everything I am responsible for, what changed overnight and what should I look at first”. No per-appliance UI answers that, and clicking through eleven of them is not a strategy.
This is a fleet sweep built on the public REST API. It polls every appliance concurrently, survives the ones that are down, and returns a single triage-ordered report. The design constraints are the interesting part: one unreachable appliance must not delay or break the whole run, and the output has to be ordered by urgency rather than alphabetically, or nobody reads past the first screen.
01. The API surface, and confirming it on your own build
Everything here uses the documented REST interface over HTTPS with a read-only account. No SSH, no support bundles, nothing that requires a maintenance window.
| Call | Path | Gives us |
|---|---|---|
| Version discovery | GET /api/info | Which API versions this appliance serves. Start here always |
| Login | POST /api/{v}/session/login | JSESSIONID. Body needs typeId = com.tintri.api.rest.vcommon.dto.rbac.RestApiCredentials |
| Appliance | GET /api/{v}/appliance/default | Model, OS version, component health, controller state |
| Alerts | GET /api/{v}/alert | Active alerts with severity and timestamps |
| Datastore | GET /api/{v}/datastore/default | Capacity, used, provisioned, dedupe and compression |
| VMs | GET /api/{v}/vm | VM count, per-VM stats, QoS |
| Service groups | GET /api/{v}/servicegroup | TGC policy groups and their members |
| Logout | GET /api/{v}/session/logout | Releases the session. Always call it |
Do not trust that list, including my spelling of it. Endpoint availability and response field names vary by TxOS build, and TGC exposes a different set from a VMstore. Run the discovery script below against one appliance of each type before you write anything that depends on a field.
#!/usr/bin/env python3
"""tintri_discover.py - establish ground truth for one appliance.
Probes which collection endpoints answer, prints the top-level keys of
each response, and reports which API versions are served. Run this once
per appliance type (VMstore and TGC) before building anything else.
Read only. Every call is a GET except the login.
"""
import argparse, json, os, sys
import requests, urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
CREDS = "com.tintri.api.rest.vcommon.dto.rbac.RestApiCredentials"
CANDIDATES = [
"/appliance/default", "/appliance", "/alert", "/datastore/default",
"/vm", "/servicegroup", "/snapshot", "/vmstore", "/user", "/role",
]
def main():
p = argparse.ArgumentParser()
p.add_argument("--host", required=True)
p.add_argument("--user", default=os.environ.get("TINTRI_USER"))
p.add_argument("--password", default=os.environ.get("TINTRI_PASS"))
p.add_argument("--version", help="skip auto-detection, e.g. v310.203")
p.add_argument("--dump", help="print the full first item of this path")
a = p.parse_args()
s = requests.Session()
s.verify = False
# --- which versions does this box actually serve?
versions = []
try:
info = s.get(f"https://{a.host}/api/info", timeout=15).json()
print("GET /api/info ->")
print(json.dumps(info, indent=2)[:900])
for key in ("supportedVersions", "versions", "preferredVersion"):
if key in info:
v = info[key]
versions += v if isinstance(v, list) else [v]
except Exception as e:
print(f"/api/info not available ({e}); will try defaults")
tries = ([a.version] if a.version
else [str(v) for v in versions] + ["v310", "v310.203", "v310.111"])
base = None
for v in tries:
v = v if str(v).startswith("v") else f"v{v}"
candidate = f"https://{a.host}/api/{v}"
r = s.post(f"{candidate}/session/login",
json={"typeId": CREDS, "username": a.user,
"password": a.password}, timeout=20)
print(f"login on {v:<12} -> HTTP {r.status_code}")
if r.status_code == 200:
base = candidate
print(f"\nUSING {base}\n")
break
if not base:
sys.exit("no working API version found; check credentials and paths")
try:
# --- which endpoints exist, and what do they return?
print(f"{'ENDPOINT':<24}{'HTTP':>6}{'ITEMS':>8} TOP-LEVEL KEYS")
print("-" * 92)
for path in CANDIDATES:
try:
r = s.get(f"{base}{path}", timeout=30)
except Exception as e:
print(f"{path:<24}{'ERR':>6}{'-':>8} {e}")
continue
if r.status_code != 200:
print(f"{path:<24}{r.status_code:>6}{'-':>8}")
continue
try:
body = r.json()
except ValueError:
print(f"{path:<24}{200:>6}{'-':>8} non-JSON response")
continue
if isinstance(body, dict) and "items" in body:
n = len(body["items"])
keys = sorted(body["items"][0].keys())[:6] if n else []
elif isinstance(body, list):
n = len(body)
keys = sorted(body[0].keys())[:6] if n else []
else:
n = 1
keys = sorted(body.keys())[:6]
print(f"{path:<24}{200:>6}{n:>8} {', '.join(keys)}")
# --- full dump of one object, so you can read real field names
if a.dump:
r = s.get(f"{base}{a.dump}", timeout=30)
body = r.json()
item = (body["items"][0] if isinstance(body, dict) and body.get("items")
else body)
print(f"\n=== FULL OBJECT FROM {a.dump} ===")
print(json.dumps(item, indent=2)[:6000])
finally:
s.get(f"{base}/session/logout", timeout=15)
if __name__ == "__main__":
main()
Run it with --dump /alert and --dump /appliance/default and keep the output. Those two objects are what the rest of this post parses, and having your build’s real field names next to you turns the following script from something you trust into something you can correct in thirty seconds.
02. Design constraints for a fleet poller
- Concurrent, with per-appliance isolation. Eleven appliances polled serially at a 30 second timeout means a five minute run if two are down. A thread pool with a hard timeout per appliance keeps the whole sweep bounded.
- An unreachable appliance is a finding, not a crash. It belongs at the top of the report, not in a stack trace.
- Ordered by urgency. Unreachable first, then critical alerts, then capacity runway, then everything healthy. Alphabetical order guarantees nobody reads past the first screen.
- Sessions always released. A sweep that runs every fifteen minutes and leaks a session each time will eventually be the reason nobody can log in.
- Credentials from the environment or a vault, never argv. Command lines end up in shell history and in the process table.
03. The sweep
#!/usr/bin/env python3
"""tintri_fleet_sweep.py - one report across every VMstore and TGC.
Collects per appliance:
* reachability and API version actually negotiated
* component and controller health
* unacknowledged alerts by severity, with age
* capacity used, provisioned, and a linear days-to-full projection
* VM count and how many are currently above a latency threshold
Concurrent with per-appliance isolation: one dead appliance neither
delays nor breaks the sweep. Read only throughout.
Usage:
export TINTRI_USER=readonly-svc TINTRI_PASS=...
./tintri_fleet_sweep.py --inventory fleet.txt
./tintri_fleet_sweep.py --inventory fleet.txt --json | jq .
fleet.txt is one host per line; blank lines and # comments ignored.
"""
import argparse, json, os, sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
import requests, urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
CREDS = "com.tintri.api.rest.vcommon.dto.rbac.RestApiCredentials"
VERSION_FALLBACKS = ["v310", "v310.203", "v310.111"]
def parse_ts(value) -> Optional[datetime]:
"""Timestamps appear as epoch millis or ISO strings depending on build."""
if value in (None, ""):
return None
if isinstance(value, (int, float)):
return datetime.fromtimestamp(value / 1000, tz=timezone.utc)
try:
dt = datetime.fromisoformat(str(value).replace("Z", "+00:00"))
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
except ValueError:
return None
def first(d: Dict[str, Any], *names, default=None):
"""Return the first present key. Field names move between builds, so
every read goes through here rather than assuming one spelling."""
for n in names:
if isinstance(d, dict) and d.get(n) not in (None, ""):
return d[n]
return default
@dataclass
class ApplianceReport:
host: str
reachable: bool = False
error: Optional[str] = None
api_version: Optional[str] = None
model: Optional[str] = None
os_version: Optional[str] = None
kind: str = "unknown" # vmstore | tgc
components_bad: List[str] = field(default_factory=list)
alerts: Dict[str, int] = field(default_factory=dict)
oldest_alert_days: Optional[float] = None
alert_examples: List[str] = field(default_factory=list)
used_pct: Optional[float] = None
used_tb: Optional[float] = None
total_tb: Optional[float] = None
provisioned_tb: Optional[float] = None
overcommit: Optional[float] = None
vm_count: Optional[int] = None
vms_over_latency: Optional[int] = None
def severity_rank(self) -> int:
"""Lower sorts first. This is the triage order."""
if not self.reachable:
return 0
if self.components_bad:
return 1
if self.alerts.get("CRITICAL", 0):
return 2
if (self.used_pct or 0) >= 90:
return 3
if self.alerts.get("WARNING", 0):
return 4
return 9
def headline(self) -> str:
if not self.reachable:
return f"UNREACHABLE {self.error}"
bits = []
if self.components_bad:
bits.append(f"components: {', '.join(self.components_bad)}")
c = self.alerts.get("CRITICAL", 0)
w = self.alerts.get("WARNING", 0)
if c:
bits.append(f"{c} critical")
if w:
bits.append(f"{w} warning")
if (self.used_pct or 0) >= 90:
bits.append(f"{self.used_pct:.0f}% full")
return "; ".join(bits) if bits else "healthy"
class Appliance:
def __init__(self, host, user, pwd, timeout=25):
self.host, self.timeout = host, timeout
self.s = requests.Session()
self.s.verify = False
self.base = None
self.version = None
self._login(user, pwd)
def _login(self, user, pwd):
last = None
for v in VERSION_FALLBACKS:
base = f"https://{self.host}/api/{v}"
try:
r = self.s.post(f"{base}/session/login",
json={"typeId": CREDS, "username": user,
"password": pwd}, timeout=self.timeout)
except requests.RequestException as e:
last = str(e)
break # transport failure: no point trying versions
if r.status_code == 200:
self.base, self.version = base, v
return
last = f"HTTP {r.status_code} on {v}"
raise RuntimeError(last or "login failed")
def get(self, path):
r = self.s.get(f"{self.base}{path}", timeout=self.timeout)
if r.status_code != 200:
return None
try:
return r.json()
except ValueError:
return None
def close(self):
try:
self.s.get(f"{self.base}/session/logout", timeout=10)
except Exception:
pass
def collect(host, user, pwd, latency_ms, timeout) -> ApplianceReport:
rep = ApplianceReport(host=host)
ap = None
try:
ap = Appliance(host, user, pwd, timeout)
rep.reachable = True
rep.api_version = ap.version
# ---------- appliance identity and component health
info = ap.get("/appliance/default") or {}
payload = info.get("items", [info])[0] if isinstance(info, dict) else {}
rep.model = first(payload, "modelName", "model", "productName")
rep.os_version = first(payload, "osVersion", "version", "softwareVersion")
rep.kind = "tgc" if str(rep.model or "").upper().startswith("TGC") \
or "GLOBAL" in str(rep.model or "").upper() else "vmstore"
# component health lives under different keys by build; walk anything
# that looks like a component list and collect non-normal states
for key in ("components", "applianceComponents", "hardwareComponents"):
for comp in (payload.get(key) or []):
if not isinstance(comp, dict):
continue
state = str(first(comp, "status", "state", "health",
default="")).upper()
name = first(comp, "name", "type", "componentType",
default="component")
if state and state not in ("NORMAL", "OK", "HEALTHY", "GOOD"):
rep.components_bad.append(f"{name}={state}")
# ---------- alerts
alerts = ap.get("/alert") or {}
items = alerts.get("items", []) if isinstance(alerts, dict) else alerts
now = datetime.now(timezone.utc)
oldest = None
for al in items or []:
if not isinstance(al, dict):
continue
ack = first(al, "acknowledged", "isAcknowledged", default=False)
if ack:
continue
sev = str(first(al, "severity", "alertSeverity",
default="UNKNOWN")).upper()
rep.alerts[sev] = rep.alerts.get(sev, 0) + 1
ts = parse_ts(first(al, "timestamp", "createTime", "lastSeen"))
if ts:
age = (now - ts).total_seconds() / 86400
oldest = age if oldest is None else max(oldest, age)
if sev in ("CRITICAL", "ERROR") and len(rep.alert_examples) < 4:
msg = first(al, "message", "description", "summary",
default="(no message)")
rep.alert_examples.append(str(msg)[:110])
rep.oldest_alert_days = round(oldest, 1) if oldest is not None else None
# ---------- capacity
ds = ap.get("/datastore/default") or {}
d = ds.get("items", [ds])[0] if isinstance(ds, dict) else {}
stat = d.get("stat") or d
used = first(stat, "spaceUsedGiB", "usedGiB", "spaceUsedPhysicalGiB")
total = first(stat, "spaceTotalGiB", "totalGiB", "capacityGiB")
prov = first(stat, "spaceProvisionedGiB", "provisionedGiB")
if used and total:
rep.used_tb = round(float(used) / 1024, 2)
rep.total_tb = round(float(total) / 1024, 2)
rep.used_pct = round(float(used) / float(total) * 100, 1)
if prov and total:
rep.provisioned_tb = round(float(prov) / 1024, 2)
rep.overcommit = round(float(prov) / float(total), 2)
# ---------- VMs, only on a VMstore
if rep.kind == "vmstore":
vms = ap.get("/vm") or {}
vitems = vms.get("items", []) if isinstance(vms, dict) else []
rep.vm_count = len(vitems)
slow = 0
for vm in vitems:
hist = (vm.get("stat") or {}).get("sortedStats") or []
if hist and float(hist[-1].get("latencyTotalMs") or 0) > latency_ms:
slow += 1
rep.vms_over_latency = slow
except Exception as e:
rep.reachable = False
rep.error = f"{type(e).__name__}: {e}"[:160]
finally:
if ap:
ap.close()
return rep
def load_inventory(path) -> List[str]:
hosts = []
with open(path) as fh:
for line in fh:
line = line.split("#", 1)[0].strip()
if line:
hosts.append(line)
return hosts
def main():
p = argparse.ArgumentParser()
p.add_argument("--inventory", required=True)
p.add_argument("--user", default=os.environ.get("TINTRI_USER"))
p.add_argument("--password", default=os.environ.get("TINTRI_PASS"))
p.add_argument("--workers", type=int, default=8)
p.add_argument("--timeout", type=int, default=25)
p.add_argument("--latency-ms", type=float, default=15.0)
p.add_argument("--capacity-warn", type=float, default=80.0)
p.add_argument("--json", action="store_true")
a = p.parse_args()
if not a.user or not a.password:
sys.exit("set TINTRI_USER and TINTRI_PASS")
hosts = load_inventory(a.inventory)
reports: List[ApplianceReport] = []
with ThreadPoolExecutor(max_workers=a.workers) as pool:
futures = {pool.submit(collect, h, a.user, a.password,
a.latency_ms, a.timeout): h for h in hosts}
for fut in as_completed(futures):
try:
reports.append(fut.result())
except Exception as e:
reports.append(ApplianceReport(host=futures[fut],
reachable=False,
error=str(e)[:160]))
reports.sort(key=lambda r: (r.severity_rank(), r.host))
if a.json:
print(json.dumps([asdict(r) for r in reports], indent=2, default=str))
return 0 if all(r.severity_rank() >= 9 for r in reports) else 2
print(f"TINTRI FLEET SWEEP {datetime.now().strftime('%Y-%m-%d %H:%M')} "
f"{len(reports)} appliances\n")
print(f"{'APPLIANCE':<30}{'KIND':<9}{'USED%':>7}{'PROV':>7}"
f"{'VMs':>6}{'SLOW':>6} STATUS")
print("-" * 110)
for r in reports:
used = f"{r.used_pct:.0f}" if r.used_pct is not None else "-"
prov = f"{r.overcommit:.1f}x" if r.overcommit else "-"
vms = str(r.vm_count) if r.vm_count is not None else "-"
slow = str(r.vms_over_latency) if r.vms_over_latency is not None else "-"
print(f"{r.host[:29]:<30}{r.kind:<9}{used:>7}{prov:>7}"
f"{vms:>6}{slow:>6} {r.headline()}")
# detail only for things that need it
needs = [r for r in reports if r.severity_rank() < 9]
if needs:
print("\nDETAIL")
for r in needs:
print(f"\n {r.host} ({r.model or '?'} / {r.os_version or '?'})")
if not r.reachable:
print(f" unreachable: {r.error}")
continue
if r.components_bad:
print(f" components : {', '.join(r.components_bad)}")
if r.alerts:
summary = ", ".join(f"{k}={v}" for k, v in
sorted(r.alerts.items()))
print(f" alerts : {summary}"
+ (f" oldest {r.oldest_alert_days}d"
if r.oldest_alert_days else ""))
for ex in r.alert_examples:
print(f" - {ex}")
if r.used_pct is not None and r.used_pct >= a.capacity_warn:
print(f" capacity : {r.used_tb} of {r.total_tb} TB "
f"({r.used_pct}%), provisioned {r.provisioned_tb} TB")
unreachable = sum(1 for r in reports if not r.reachable)
critical = sum(r.alerts.get("CRITICAL", 0) for r in reports)
print(f"\nunreachable {unreachable} critical alerts {critical} "
f"needing attention {len(needs)} of {len(reports)}")
return 0 if not needs else 2
if __name__ == "__main__":
sys.exit(main())
04. Two details in that code worth calling out
Every field read goes through first()
Field names move between TxOS builds. Rather than pick one spelling and have the script silently report zeros on half the fleet, every read tries several candidates and falls back to a default. If your discovery output shows a name I did not list, add it to the tuple and the whole script picks it up. This is the difference between a script that works on one appliance and one that works on the fleet you actually have, which is rarely all on the same version.
Version fallback stops on a transport error
If login returns HTTP 404 the script tries the next version path, because that is the fully-qualified-version problem. If the connection itself fails it stops immediately rather than retrying three times, because an unreachable appliance is unreachable on every version and retrying just triples the time the sweep spends on a dead box.
05. Reading the report
| What you see | What it means | Do this |
|---|---|---|
| UNREACHABLE | Login or transport failed | Check management network and credentials before assuming an outage. A rotated service account password presents identically |
| components listed | Hardware or controller not in a normal state | Go to the appliance directly. This outranks every alert |
| oldest alert over 30 days | Alerts are being generated and nobody acknowledges them | A process problem, not a technical one. Unacknowledged noise is why real alerts get missed |
| PROV above about 3x | Heavy thin provisioning | Not wrong in itself, but it means used percent understates your exposure if the guests fill up |
| USED above 90 | Capacity runway is short | Check snapshot-held space first. It is usually the cheapest reclaim |
| SLOW count high on one appliance only | Localised, not fleet-wide | Per-VM breakdown on that appliance. Fleet-wide slowness is almost never the array |
The row that most often turns out to matter is the oldest unacknowledged alert. An appliance carrying forty warnings, the oldest ninety days old, is not an appliance with forty problems. It is an appliance nobody is watching, and the real risk is the critical alert that will arrive next week and be invisible in the noise.
06. PowerShell, for the interactive case
When you are on a call and want the same answer without a Python environment, this covers the same ground using Invoke-RestMethod and PowerShell 7 parallel processing.
#requires -Version 7.0
<#
tintri-fleet-sweep.ps1
Fleet health across every VMstore and TGC listed in -Inventory.
Read only. Sessions are released in a finally block.
#>
param(
[Parameter(Mandatory)][string]$Inventory,
[Parameter(Mandatory)][pscredential]$Credential,
[int]$ThrottleLimit = 8,
[double]$CapacityWarn = 80
)
$hosts = Get-Content $Inventory |
ForEach-Object { ($_ -split '#')[0].Trim() } |
Where-Object { $_ }
$user = $Credential.UserName
$pass = $Credential.GetNetworkCredential().Password
$results = $hosts | ForEach-Object -ThrottleLimit $ThrottleLimit -Parallel {
$h = $_
$user = $using:user
$pass = $using:pass
$creds = 'com.tintri.api.rest.vcommon.dto.rbac.RestApiCredentials'
$row = [ordered]@{
Appliance = $h; Reachable = $false; Model = $null; OS = $null
UsedPct = $null; ProvX = $null; VMs = $null
Critical = 0; Warning = 0; Components = ''; Note = ''
}
$session = $null
foreach ($v in @('v310','v310.203','v310.111')) {
$base = "https://$h/api/$v"
try {
$body = @{ typeId = $creds; username = $user; password = $pass } |
ConvertTo-Json
Invoke-RestMethod -Method Post -Uri "$base/session/login" `
-Body $body -ContentType 'application/json' `
-SkipCertificateCheck -SessionVariable ws -TimeoutSec 25 | Out-Null
$session = $ws; $row.Reachable = $true
break
}
catch {
$code = $_.Exception.Response.StatusCode.value__
if ($code -ne 404) { $row.Note = $_.Exception.Message; break }
}
}
if (-not $session) {
if (-not $row.Note) { $row.Note = 'login failed on all versions' }
return [pscustomobject]$row
}
try {
$g = { param($p) Invoke-RestMethod -Uri "$base$p" -WebSession $session `
-SkipCertificateCheck -TimeoutSec 25 }
$app = & $g '/appliance/default'
$a0 = if ($app.items) { $app.items[0] } else { $app }
$row.Model = $a0.modelName ?? $a0.model
$row.OS = $a0.osVersion ?? $a0.version
$bad = foreach ($c in @($a0.components)) {
$st = ($c.status ?? $c.state ?? '').ToString().ToUpper()
if ($st -and $st -notin 'NORMAL','OK','HEALTHY','GOOD') {
"$($c.name ?? $c.type)=$st"
}
}
$row.Components = ($bad -join ', ')
$alerts = & $g '/alert'
foreach ($al in @($alerts.items)) {
if ($al.acknowledged) { continue }
switch (($al.severity ?? '').ToString().ToUpper()) {
'CRITICAL' { $row.Critical++ }
'ERROR' { $row.Critical++ }
'WARNING' { $row.Warning++ }
}
}
$ds = & $g '/datastore/default'
$d = if ($ds.items) { $ds.items[0] } else { $ds }
$st = $d.stat ?? $d
$used = $st.spaceUsedGiB ?? $st.usedGiB
$total = $st.spaceTotalGiB ?? $st.totalGiB
$prov = $st.spaceProvisionedGiB ?? $st.provisionedGiB
if ($used -and $total) { $row.UsedPct = [math]::Round($used/$total*100,1) }
if ($prov -and $total) { $row.ProvX = [math]::Round($prov/$total,1) }
try { $row.VMs = @((& $g '/vm').items).Count } catch { }
}
catch { $row.Note = $_.Exception.Message }
finally {
try { & $g '/session/logout' | Out-Null } catch { }
}
[pscustomobject]$row
}
# triage order, worst first
$ranked = $results | Sort-Object `
@{ Expression = {
if (-not $_.Reachable) { 0 }
elseif ($_.Components) { 1 }
elseif ($_.Critical -gt 0) { 2 }
elseif ($_.UsedPct -ge 90) { 3 }
elseif ($_.Warning -gt 0) { 4 }
else { 9 }
}}, Appliance
$ranked | Format-Table Appliance, Model, UsedPct, ProvX, VMs,
Critical, Warning, Components, Note -AutoSize
$down = @($ranked | Where-Object { -not $_.Reachable }).Count
$crit = ($ranked | Measure-Object Critical -Sum).Sum
Write-Host "`nunreachable $down critical alerts $crit appliances $($ranked.Count)"
if ($down -or $crit) { exit 2 }
Both scripts exit 2 when anything needs attention and 0 when the fleet is clean, so either drops into cron or a scheduled task that only mails you when there is something to read.
More in this series to follow: capacity runway with snapshot-held space broken out, and replication lag across a TGC-managed estate.
Endpoint availability and field names vary by TxOS build and differ between VMstore and TGC. Run the discovery script against one of each before relying on any field, and extend the candidate lists in first() to match what you find. A read-only service account is sufficient throughout. Nothing here is official guidance from any vendor.
Leave a Reply