Part 1 of 10. SDDC Manager is not a view onto vCenter. It maintains its own inventory database, populated at commission time and updated only by its own workflows. vCenter maintains a different one. Nothing continuously reconciles them.
When they diverge you do not get an error. You get a workflow six weeks later that fails on a precondition check, with a message naming a component that is working perfectly. This post is about detecting that divergence before it costs you a maintenance window, and about the three API behaviours that make naive scripts report the wrong answer confidently.
01. Three API behaviours that will silently corrupt your results
Pagination, which is the one that bites hardest
Collection endpoints return a page, not a collection. The response carries a pageMetadata block, and the elements array holds only the current page. A script that reads .elements once and stops gets the first page and reports it as the whole fleet. On a small lab this is invisible, because everything fits in one page. On a real estate it means your audit quietly ignores most of your hosts.
{
"elements": [ ... ],
"pageMetadata": {
"pageNumber": 0,
"pageSize": 100,
"totalElements": 247, <-- you have 247, you fetched 100
"totalPages": 3
}
}Token expiry mid-run
The access token is short lived. A fleet-wide audit that walks every host and cluster can outlive it. The failure is a 401 partway through a loop, which an unguarded script turns into a stack trace after twenty minutes of work. Worse, a script that catches and ignores exceptions turns it into a partial result that looks complete.
Transient 5xx during background activity
SDDC Manager runs its own scheduled work. If your audit lands during a bundle download or an inventory sync you will see intermittent 502 and 503. These are transient and retryable, and a script without backoff reports them as findings.
All three produce a plausible wrong answer rather than an obvious failure. That is what makes them dangerous in an audit script. The session class below handles all three, and every later part of this series reuses it.
02. A session class that is actually safe to schedule
#!/usr/bin/env python3
"""vcf_session.py - a reusable SDDC Manager client.
Handles the three failure modes that corrupt naive audits:
* paginated collections -> paginate() exhausts every page
* access token expiry mid-run -> transparent refresh on 401, once
* transient 5xx from background -> exponential backoff with jitter
work on the appliance
"""
import logging
import random
import threading
import time
from typing import Iterator, Optional
import requests
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
log = logging.getLogger("vcf")
RETRYABLE = {429, 500, 502, 503, 504}
class VcfError(RuntimeError):
"""Non-retryable API failure, carrying the server's own message."""
def __init__(self, status: int, path: str, body: str):
self.status, self.path = status, path
super().__init__(f"HTTP {status} on {path}: {body[:400]}")
class VcfSession:
def __init__(self, host, username, password,
verify=False, timeout=60, max_retries=4):
self.base = f"https://{host}/v1"
self.timeout = timeout
self.max_retries = max_retries
self._creds = (username, password)
self._lock = threading.Lock() # guards token refresh
self._refresh_id: Optional[str] = None
self.s = requests.Session()
self.s.verify = verify
self._authenticate()
# ---------- authentication ----------
def _authenticate(self):
user, pwd = self._creds
r = self.s.post(f"{self.base}/tokens",
json={"username": user, "password": pwd},
timeout=self.timeout)
if r.status_code >= 400:
raise VcfError(r.status_code, "/tokens", r.text)
data = r.json()
self._apply(data["accessToken"])
# refreshToken is an object in most releases, a string in some
rt = data.get("refreshToken")
self._refresh_id = rt.get("id") if isinstance(rt, dict) else rt
log.debug("authenticated, refresh token %s",
"present" if self._refresh_id else "absent")
def _apply(self, token: str):
self.s.headers["Authorization"] = f"Bearer {token}"
def _renew(self):
"""Refresh the access token. Falls back to a full re-auth.
Serialised so that concurrent workers hitting 401 at the same
moment do not each burn a refresh.
"""
with self._lock:
if self._refresh_id:
r = self.s.patch(
f"{self.base}/tokens/access-token/refresh",
data=self._refresh_id,
headers={"Content-Type": "application/json"},
timeout=self.timeout)
if r.status_code < 400:
self._apply(r.json()["accessToken"])
log.debug("access token refreshed")
return
log.warning("refresh failed (%s), re-authenticating",
r.status_code)
self._authenticate()
# ---------- transport ----------
def request(self, method: str, path: str, **kw):
url = f"{self.base}{path}"
refreshed = False
for attempt in range(self.max_retries + 1):
try:
r = self.s.request(method, url, timeout=self.timeout, **kw)
except requests.RequestException as e:
if attempt == self.max_retries:
raise
self._sleep(attempt, f"transport error: {e}")
continue
if r.status_code == 401 and not refreshed:
# one refresh per call, then treat 401 as terminal
refreshed = True
self._renew()
continue
if r.status_code in RETRYABLE and attempt < self.max_retries:
self._sleep(attempt, f"HTTP {r.status_code}")
continue
if r.status_code >= 400:
raise VcfError(r.status_code, path, r.text)
return r.json() if r.content else {}
raise VcfError(0, path, "retries exhausted")
def _sleep(self, attempt: int, why: str):
# exponential backoff, jittered so parallel runs do not resonate
delay = min(2 ** attempt, 16) + random.uniform(0, 0.75)
log.warning("%s, retry %d in %.1fs", why, attempt + 1, delay)
time.sleep(delay)
def get(self, path: str, **kw):
return self.request("GET", path, **kw)
# ---------- pagination ----------
def paginate(self, path: str, page_size: int = 100) -> Iterator[dict]:
"""Yield every element across every page.
Falls back gracefully on endpoints that ignore paging params and
return a bare list, which several VCF endpoints still do.
"""
page = 0
seen = 0
while True:
sep = "&" if "?" in path else "?"
body = self.get(f"{path}{sep}pageNumber={page}&pageSize={page_size}")
if isinstance(body, list):
yield from body
return
elements = body.get("elements")
if elements is None:
yield body
return
yield from elements
seen += len(elements)
meta = body.get("pageMetadata") or {}
total_pages = meta.get("totalPages")
total = meta.get("totalElements")
if total_pages is None:
# no metadata: stop when a short page comes back
if len(elements) < page_size:
return
elif page + 1 >= total_pages:
if total is not None and seen != total:
log.warning("%s: collected %d of %d elements",
path, seen, total)
return
page += 103. The reconciliation itself
Now the actual question. Take the set of hosts SDDC Manager believes exist, take the set vCenter believes exist, and compare them in both directions. The interesting findings are the asymmetries.
#!/usr/bin/env python3
"""vcf_inventory_drift.py - reconcile SDDC Manager against live vCenter."""
import argparse
import logging
import ssl
import sys
from collections import defaultdict
from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
from vcf_session import VcfSession, VcfError
logging.basicConfig(level=logging.WARNING,
format="%(levelname)s %(message)s")
def sddc_inventory(vcf):
"""What SDDC Manager believes: hosts, their domain, cluster and state."""
domains = {d["id"]: d for d in vcf.paginate("/domains")}
clusters = {c["id"]: c for c in vcf.paginate("/clusters")}
hosts = {}
for h in vcf.paginate("/hosts"):
fqdn = (h.get("fqdn") or "").lower()
if not fqdn:
continue
cid = (h.get("cluster") or {}).get("id")
did = (h.get("domain") or {}).get("id")
hosts[fqdn] = {
"id": h.get("id"),
"status": h.get("status"),
"storage": h.get("storageType"),
"version": h.get("esxiVersion") or h.get("version"),
"cluster": (clusters.get(cid) or {}).get("name"),
"domain": (domains.get(did) or {}).get("name"),
}
return domains, clusters, hosts
def vcenter_inventory(host, user, pwd):
"""What vCenter believes: connected hosts, build, cluster, maintenance."""
ctx = ssl._create_unverified_context()
si = SmartConnect(host=host, user=user, pwd=pwd, sslContext=ctx)
try:
content = si.RetrieveContent()
view = content.viewManager.CreateContainerView(
content.rootFolder, [vim.HostSystem], True)
out = {}
for h in view.view:
parent = h.parent
out[h.name.lower()] = {
"state": h.runtime.connectionState,
"maintenance": h.runtime.inMaintenanceMode,
"build": h.config.product.build if h.config else None,
"version": h.config.product.version if h.config else None,
"cluster": parent.name if isinstance(
parent, vim.ClusterComputeResource) else None,
}
view.Destroy()
return out
finally:
Disconnect(si)
def reconcile(sddc_hosts, vc_hosts):
findings = defaultdict(list)
only_sddc = set(sddc_hosts) - set(vc_hosts)
only_vc = set(vc_hosts) - set(sddc_hosts)
both = set(sddc_hosts) & set(vc_hosts)
for fqdn in sorted(only_sddc):
findings["ORPHAN_IN_SDDC"].append(
f"{fqdn} is in SDDC Manager inventory "
f"(status={sddc_hosts[fqdn]['status']}) but absent from vCenter")
for fqdn in sorted(only_vc):
findings["UNMANAGED_IN_VC"].append(
f"{fqdn} exists in vCenter "
f"(cluster={vc_hosts[fqdn]['cluster']}) but SDDC Manager "
"has no record of it")
for fqdn in sorted(both):
s, v = sddc_hosts[fqdn], vc_hosts[fqdn]
if s["cluster"] and v["cluster"] and s["cluster"] != v["cluster"]:
findings["CLUSTER_MISMATCH"].append(
f"{fqdn}: SDDC Manager says {s['cluster']}, "
f"vCenter says {v['cluster']}")
if v["state"] != "connected":
findings["HOST_NOT_CONNECTED"].append(
f"{fqdn}: vCenter reports {v['state']}")
if v["maintenance"]:
findings["IN_MAINTENANCE"].append(
f"{fqdn}: still in maintenance mode")
if s["version"] and v["version"] and not \
str(s["version"]).startswith(str(v["version"])):
findings["VERSION_MISMATCH"].append(
f"{fqdn}: SDDC Manager recorded {s['version']}, "
f"host is running {v['version']} build {v['build']}")
return findings
def stuck_tasks(vcf, hours=24):
"""In-progress tasks older than N hours block later workflows."""
out = []
for t in vcf.paginate("/tasks"):
if t.get("status") in ("IN_PROGRESS", "Pending"):
out.append(f"{t.get('name')} [{t.get('id')}] "
f"status={t.get('status')} "
f"started={t.get('creationTimestamp')}")
elif t.get("status") == "Failed":
out.append(f"FAILED: {t.get('name')} [{t.get('id')}]")
return out
def main():
p = argparse.ArgumentParser()
p.add_argument("--sddc", required=True)
p.add_argument("--sddc-user", required=True)
p.add_argument("--sddc-pass", required=True)
p.add_argument("--vcenter", required=True)
p.add_argument("--vc-user", required=True)
p.add_argument("--vc-pass", required=True)
p.add_argument("-v", "--verbose", action="store_true")
a = p.parse_args()
if a.verbose:
logging.getLogger("vcf").setLevel(logging.DEBUG)
try:
vcf = VcfSession(a.sddc, a.sddc_user, a.sddc_pass)
except VcfError as e:
sys.exit(f"cannot reach SDDC Manager: {e}")
domains, clusters, sddc_hosts = sddc_inventory(vcf)
vc_hosts = vcenter_inventory(a.vcenter, a.vc_user, a.vc_pass)
print(f"SDDC Manager : {len(domains)} domains, "
f"{len(clusters)} clusters, {len(sddc_hosts)} hosts")
print(f"vCenter : {len(vc_hosts)} hosts\n")
findings = reconcile(sddc_hosts, vc_hosts)
if not findings:
print("Inventories agree.")
for kind in sorted(findings):
print(f"[{kind}]")
for line in findings[kind]:
print(f" {line}")
print()
tasks = stuck_tasks(vcf)
if tasks:
print("[TASKS] in-progress or failed, these block later workflows")
for t in tasks:
print(f" {t}")
return 2 if (findings or tasks) else 0
if __name__ == "__main__":
sys.exit(main())04. What each finding actually means
| Finding | Usual cause | What it breaks later | Action |
|---|---|---|---|
| ORPHAN_IN_SDDC | Host removed from vCenter directly, or a decommission that failed after the vCenter step | Upgrade prechecks fail trying to contact a host that is gone. Capacity reporting overstates the fleet | Complete the decommission through SDDC Manager rather than deleting again in vCenter |
| UNMANAGED_IN_VC | Host added straight into a VCF-managed cluster in the vSphere Client | The host receives no lifecycle updates and is invisible to VCF workflows. On a vSAN cluster it can affect the cluster while VCF has no record of it | Remove it from the cluster and commission it properly, or accept it is out of band and document it |
| CLUSTER_MISMATCH | Host moved between clusters in vCenter | Workload domain operations target the wrong cluster. This is the one that causes genuinely surprising failures | Move it back, then perform the move through VCF |
| VERSION_MISMATCH | Host patched out of band | Upgrade planning is computed from stale data. The host may be outside the BOM entirely | Re-run inventory sync, and confirm the build against your VCF BOM |
| HOST_NOT_CONNECTED | Genuine outage, or a certificate trust failure | Everything | If several appeared at once after a certificate change, see part 8 |
| TASKS | A workflow that never completed | A stuck task can hold a lock that makes the next operation fail with an unrelated message | Resolve or explicitly fail the task before starting anything else |
The general rule behind every row: make the change through SDDC Manager, or SDDC Manager will not know about it. The vSphere Client will happily let you move, patch or remove a VCF-managed host, and nothing warns you at the time. The cost arrives at the next upgrade.
05. PowerShell, same reconciliation
#requires -Version 7.0
#requires -Modules VMware.PowerCLI
<#
vcf-inventory-drift.ps1
Reconciles SDDC Manager inventory against live vCenter state.
Handles pagination and token expiry, which the obvious version does not.
#>
param(
[Parameter(Mandatory)][string]$SddcManager,
[Parameter(Mandatory)][pscredential]$SddcCredential,
[Parameter(Mandatory)][string]$VCenter,
[Parameter(Mandatory)][pscredential]$VcCredential
)
$ErrorActionPreference = 'Stop'
$script:base = "https://$SddcManager/v1"
$script:token = $null
$script:refresh = $null
$skip = @{ SkipCertificateCheck = $true }
function Connect-Vcf {
$body = @{
username = $SddcCredential.UserName
password = $SddcCredential.GetNetworkCredential().Password
} | ConvertTo-Json
$r = Invoke-RestMethod -Method Post -Uri "$script:base/tokens" `
-Body $body -ContentType 'application/json' @skip
$script:token = $r.accessToken
$script:refresh = if ($r.refreshToken -is [string]) { $r.refreshToken }
else { $r.refreshToken.id }
}
function Invoke-Vcf {
param([string]$Path)
$hdr = @{ Authorization = "Bearer $script:token" }
try {
Invoke-RestMethod -Method Get -Uri "$script:base$Path" -Headers $hdr @skip
}
catch {
if ($_.Exception.Response.StatusCode.value__ -eq 401) {
Connect-Vcf # token aged out mid-run
$hdr = @{ Authorization = "Bearer $script:token" }
Invoke-RestMethod -Method Get -Uri "$script:base$Path" -Headers $hdr @skip
}
else { throw }
}
}
function Get-VcfAll {
# exhausts every page rather than returning page zero
param([string]$Path, [int]$PageSize = 100)
$page = 0
do {
$sep = if ($Path -match '\?') { '&' } else { '?' }
$r = Invoke-Vcf "$Path$sep`pageNumber=$page&pageSize=$PageSize"
if ($r -is [array]) { $r; return }
if ($null -eq $r.elements) { $r; return }
$r.elements
$total = $r.pageMetadata.totalPages
$page++
} while ($total -and $page -lt $total)
}
Connect-Vcf
$domains = @{}; Get-VcfAll '/domains' | ForEach-Object { $domains[$_.id] = $_ }
$clusters = @{}; Get-VcfAll '/clusters' | ForEach-Object { $clusters[$_.id] = $_ }
$sddc = @{}
foreach ($h in (Get-VcfAll '/hosts')) {
if (-not $h.fqdn) { continue }
$sddc[$h.fqdn.ToLower()] = [pscustomobject]@{
Status = $h.status
Storage = $h.storageType
Version = $h.esxiVersion
Cluster = $clusters[$h.cluster.id].name
Domain = $domains[$h.domain.id].name
}
}
Connect-VIServer -Server $VCenter -Credential $VcCredential | Out-Null
try {
$vc = @{}
foreach ($h in Get-VMHost) {
$vc[$h.Name.ToLower()] = [pscustomobject]@{
State = $h.ConnectionState
Maintenance = ($h.ConnectionState -eq 'Maintenance')
Version = $h.Version
Build = $h.Build
Cluster = (Get-Cluster -VMHost $h -ErrorAction SilentlyContinue).Name
}
}
}
finally { Disconnect-VIServer -Confirm:$false -ErrorAction SilentlyContinue }
"SDDC Manager : {0} domains, {1} clusters, {2} hosts" -f `
$domains.Count, $clusters.Count, $sddc.Count
"vCenter : {0} hosts`n" -f $vc.Count
$issues = 0
foreach ($f in ($sddc.Keys | Where-Object { $_ -notin $vc.Keys } | Sort-Object)) {
Write-Host "[ORPHAN_IN_SDDC] $f (status=$($sddc[$f].Status))" -ForegroundColor Red
$issues++
}
foreach ($f in ($vc.Keys | Where-Object { $_ -notin $sddc.Keys } | Sort-Object)) {
Write-Host "[UNMANAGED_IN_VC] $f (cluster=$($vc[$f].Cluster))" -ForegroundColor Red
$issues++
}
foreach ($f in ($sddc.Keys | Where-Object { $_ -in $vc.Keys } | Sort-Object)) {
$s = $sddc[$f]; $v = $vc[$f]
if ($s.Cluster -and $v.Cluster -and $s.Cluster -ne $v.Cluster) {
Write-Host "[CLUSTER_MISMATCH] $f SDDC=$($s.Cluster) vCenter=$($v.Cluster)" -ForegroundColor Yellow
$issues++
}
if ($v.State -ne 'Connected') {
Write-Host "[NOT_CONNECTED] $f $($v.State)" -ForegroundColor Yellow
$issues++
}
if ($s.Version -and $v.Version -and -not $s.Version.StartsWith($v.Version)) {
Write-Host "[VERSION_MISMATCH] $f recorded=$($s.Version) actual=$($v.Version) build $($v.Build)" -ForegroundColor Yellow
$issues++
}
}
$open = @(Get-VcfAll '/tasks' | Where-Object { $_.status -in 'IN_PROGRESS','Pending','Failed' })
foreach ($t in $open) {
Write-Host "[TASK] $($t.status): $($t.name) [$($t.id)]" -ForegroundColor Magenta
$issues++
}
Write-Host "`nFindings: $issues"
if ($issues) { exit 2 }06. Running it usefully
Both scripts exit 2 on findings and 0 on a clean run, so they drop straight into a pipeline or a cron job that only mails you when something is wrong. Run it weekly, and run it immediately before any upgrade. The prechecks you are about to run will consult the same inventory, and it is considerably cheaper to find the drift now than to find it when the upgrade aborts at 40 percent.
Next in the series: per-VM latency from the Tintri VMstore API, and the four-way split that ends most storage arguments in a single call.
Response field names vary across VCF releases, particularly esxiVersion, storageType and the shape of refreshToken. Dump one host object and confirm before scheduling. Read only credentials are sufficient on both sides. Nothing here is official guidance.
Leave a Reply