Configuration maximums tell you what is supported. They do not tell you what performs, and at scale the difference between those two numbers is where architects get hurt. A vCenter can be within every published maximum and still take four minutes to answer an inventory query, because the limit that bit was never in the table.
This is about the constraints that emerge somewhere past a few thousand managed objects: how the API actually retrieves inventory, what the SSO topology costs you when a node is lost, and how long a maintenance window really takes. Each section ends with something you can measure on your own estate rather than take on faith.
01. The limits that bind before the documented ones
| Documented limit | What actually binds first | Symptom when it does |
|---|---|---|
| Hosts per vCenter | Task and event throughput during concurrent operations | Tasks queue, the client shows spinners, automation times out |
| VMs per vCenter | Property Collector round trips from badly written clients | vpxd CPU pegged by monitoring tools, not by users |
| VMs per cluster | HA restart concurrency and storage boot storm | Restart takes far longer than the RTO assumed |
| Hosts per cluster | Maintenance mode evacuation time | The patch window is exceeded and rollback is not an option |
| Sessions | Scripts that never call logout | New logins refused; nobody can connect at all |
| Linked vCenters | Replication convergence after an outage | Permissions and tags inconsistent across the estate |
Every row is a design decision, not an operational one. You cannot fix a Property Collector problem by adding vCPUs to vCenter, and you cannot fix an evacuation-time problem during the outage it causes. These are sized in the design phase or paid for later.
02. The Property Collector, and why most monitoring tools are the load
This is the single most consequential API decision in a large estate, and it is almost absent from operational documentation. The vSphere API does not work the way most scripts assume.
A managed object reference is a handle, not an object. Reading a property from it is a remote call. So the familiar pattern below issues one round trip per property per object, and its cost is the product of your inventory size and how many fields you touch.
# THE NAIVE PATTERN. Correct output, pathological cost at scale.
view = content.viewManager.CreateContainerView(
content.rootFolder, [vim.VirtualMachine], True)
for vm in view.view: # 8000 handles
print(vm.name, # round trip
vm.runtime.powerState, # round trip
vm.config.hardware.memoryMB, # round trip
vm.guest.hostName) # round trip
# 8000 VMs x 4 properties = 32,000 sequential round trips.
# At even 3 ms each that is roughly 96 seconds of pure latency,
# and vpxd is doing real work for every one of them.
#
# Now run this every 60 seconds from a monitoring system.
# Congratulations: your monitoring is the largest consumer
# of your management plane.The Property Collector exists precisely to avoid this. You describe the traversal and the exact properties you want, and the server returns everything in a small number of paged responses.
The API objects involved
| Object | Role |
|---|---|
PropertyCollector.RetrievePropertiesEx | Bulk retrieval. Returns a page plus a continuation token |
PropertyCollector.ContinueRetrievePropertiesEx | Fetches the next page using that token |
PropertyCollector.WaitForUpdatesEx | Change notification. Returns only what changed since a version token |
PropertyFilterSpec | The query. Combines an object set and a property set |
ObjectSpec plus TraversalSpec | Where to start and how to walk the inventory tree |
PropertySpec.pathSet | Exactly which properties, by dotted path. This is the cost control |
RetrieveOptions.maxObjects | Page size. Without it a large estate builds one enormous response |
#!/usr/bin/env python3
"""vsphere_bulk.py - Property Collector retrieval that scales.
One PropertyFilterSpec, paged, returning only the named properties.
Use this shape in anything that runs on a schedule.
"""
import ssl
from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim, vmodl
def bulk_properties(si, obj_type, paths, page=250, root=None):
"""Retrieve `paths` for every object of `obj_type`.
Returns {moref: {path: value}}. Cost is O(pages), not O(objects x paths).
"""
content = si.RetrieveContent()
pc = content.propertyCollector
root = root or content.rootFolder
# a container view is still the cheapest way to scope the traversal;
# the saving comes from not dereferencing each handle afterwards
view = content.viewManager.CreateContainerView(root, [obj_type], True)
try:
# walk from the view to its 'view' property, which holds the objects
traversal = vmodl.query.PropertyCollector.TraversalSpec(
name="viewToObject", type=vim.view.ContainerView,
path="view", skip=False)
obj_spec = vmodl.query.PropertyCollector.ObjectSpec(
obj=view, skip=True, selectSet=[traversal])
prop_spec = vmodl.query.PropertyCollector.PropertySpec(
type=obj_type, all=False, pathSet=list(paths))
filter_spec = vmodl.query.PropertyCollector.FilterSpec(
objectSet=[obj_spec], propSet=[prop_spec])
opts = vmodl.query.PropertyCollector.RetrieveOptions(maxObjects=page)
results = {}
batch = pc.RetrievePropertiesEx(specSet=[filter_spec], options=opts)
pages = 0
while batch:
pages += 1
for o in batch.objects:
row = {}
for p in (o.propSet or []):
row[p.name] = p.val
# missingSet matters: a permissions gap or a VM mid-delete
# shows up here rather than raising
for m in (o.missingSet or []):
row[m.path] = None
results[o.obj] = row
if not batch.token:
break
batch = pc.ContinueRetrievePropertiesEx(token=batch.token)
return results, pages
finally:
view.Destroy()
def main():
ctx = ssl._create_unverified_context()
si = SmartConnect(host="vcenter.lab.local", user="ro@vsphere.local",
pwd="REDACTED", sslContext=ctx)
try:
rows, pages = bulk_properties(
si, vim.VirtualMachine,
paths=["name",
"runtime.powerState",
"runtime.host",
"config.hardware.memoryMB",
"config.hardware.numCPU",
"summary.quickStats.balloonedMemory"],
page=250)
print(f"{len(rows)} VMs retrieved in {pages} pages")
finally:
Disconnect(si)
if __name__ == "__main__":
main()Prove the difference on your own vCenter
Do not take the argument on trust. This harness times both approaches against the same inventory and prints the ratio. Run it out of hours, because the naive path is genuinely expensive.
#!/usr/bin/env python3
"""pc_benchmark.py - measure naive traversal against Property Collector.
Run with a small --limit first. The naive path is the load you are
trying to prove is unacceptable, so do not unleash it on production
at full inventory during business hours.
"""
import argparse, ssl, time
from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
from vsphere_bulk import bulk_properties
PATHS = ["name", "runtime.powerState",
"config.hardware.memoryMB", "config.hardware.numCPU"]
def naive(si, limit):
content = si.RetrieveContent()
view = content.viewManager.CreateContainerView(
content.rootFolder, [vim.VirtualMachine], True)
try:
t0 = time.perf_counter()
n = 0
for vm in view.view[:limit]:
_ = (vm.name,
vm.runtime.powerState,
vm.config.hardware.memoryMB,
vm.config.hardware.numCPU)
n += 1
return time.perf_counter() - t0, n
finally:
view.Destroy()
def main():
p = argparse.ArgumentParser()
p.add_argument("--vcenter", required=True)
p.add_argument("--user", required=True)
p.add_argument("--password", required=True)
p.add_argument("--limit", type=int, default=200,
help="how many VMs the NAIVE path touches")
a = p.parse_args()
ctx = ssl._create_unverified_context()
si = SmartConnect(host=a.vcenter, user=a.user, pwd=a.password,
sslContext=ctx)
try:
t_naive, n = naive(si, a.limit)
t0 = time.perf_counter()
rows, pages = bulk_properties(si, vim.VirtualMachine, PATHS)
t_pc = time.perf_counter() - t0
per_vm_naive = t_naive / max(n, 1)
per_vm_pc = t_pc / max(len(rows), 1)
print(f"naive : {n:6d} VMs in {t_naive:7.2f}s "
f"({per_vm_naive*1000:6.1f} ms/VM)")
print(f"property collector: {len(rows):6d} VMs in {t_pc:7.2f}s "
f"({per_vm_pc*1000:6.1f} ms/VM, {pages} pages)")
print(f"ratio per VM : {per_vm_naive/max(per_vm_pc,1e-9):.1f}x")
print(f"\nextrapolated to full inventory ({len(rows)} VMs):")
print(f" naive would take ~{per_vm_naive*len(rows):.0f}s")
print(f" every polling cycle, against vpxd")
finally:
Disconnect(si)
if __name__ == "__main__":
main()The design conclusion. In a large estate, mandate Property Collector usage in anything that polls. Then go and audit what your existing monitoring, backup and CMDB integrations actually do, because at least one of them is almost certainly running the naive pattern on a one minute timer. That is a capacity decision disguised as a tooling decision.
For continuous monitoring, do not poll at all
If a tool needs current state continuously, WaitForUpdatesEx is the correct primitive. Register a filter once, then block on the call; the server returns only what changed, with a new version token. Cost becomes proportional to the rate of change rather than to inventory size, which is a fundamentally better scaling property.
# conceptual shape; the filter is created once and reused
pc = content.propertyCollector
pc_filter = pc.CreateFilter(spec=filter_spec, partialUpdates=True)
version = ""
while True:
update = pc.WaitForUpdatesEx(
version=version,
options=vmodl.query.PropertyCollector.WaitOptions(
maxWaitSeconds=60)) # long poll, returns early on change
if update is None:
continue # nothing changed within the window
version = update.version
for fs in update.filterSet:
for os_ in fs.objectSet:
# os_.kind is 'enter', 'modify' or 'leave'
handle(os_.obj, os_.kind, os_.changeSet)
# 8000 VMs at rest cost nothing.
# 12 VMs power on: you receive 12 change records.03. SSO topology: the decision you cannot easily reverse
Multiple vCenters in one SSO domain give you Enhanced Linked Mode: a single pane, shared tags, shared roles, cross-vCenter migration. They also give you a replication relationship, and that is a shared failure domain most designs do not analyse.
OPTION 1 one SSO domain, all vCenters linked
+----------+ replication +----------+ replication +----------+
| vc-lon-1 |<===============>| vc-lon-2 |<===============>| vc-fra-1 |
+----------+ +----------+ +----------+
Gains single pane, shared tags and roles, xVC-vMotion
Costs one identity fault domain. A bad permission change or a
corrupted replication state propagates everywhere
Watch ring vs star topology. A star with a single hub means the
hub is a single point of partition for the whole estate
OPTION 2 separate SSO domains per region
+----------+ +----------+
| vc-lon-1 | | vc-fra-1 |
+----------+ +----------+
(isolated) (isolated)
Gains true blast radius isolation. A regional failure is regional
Costs no shared tags, no shared roles, no native xVC-vMotion,
duplicated identity administration forever
The question that decides it:
Do you migrate VMs BETWEEN these vCenters as normal operations?
yes -> one domain, and design the replication topology carefully
no -> separate domains, and stop paying for a feature you never useThe failure mode worth designing against is a replication partition rather than a node loss. If two vCenters in one domain cannot replicate but both remain up, both continue serving, and configuration changes made on each side diverge. Convergence when the link returns is not always clean. Monitor replication status explicitly; it is not covered by a vCenter health check that only reports service state.
| Design decision | Justification | Rejected alternative | Risk accepted |
|---|---|---|---|
| One SSO domain per region, not globally | xVC-vMotion is used within a region and never across regions | Single global domain. Gives one pane but makes identity a global fault domain for no operational gain | Tags and roles must be maintained per region. Automated with config as code |
| Ring replication, not star | No single node whose loss partitions the rest | Star. Simpler to reason about, but the hub is a single point of partition | Convergence is slower across a ring |
| Dedicated read-only service accounts per integration | Session exhaustion and audit attribution | One shared automation account. Simpler, but one leaking script blocks every integration and the log shows one identity | More accounts to rotate |
04. Evacuation arithmetic: how long the window actually is
Patching a large cluster is bounded by vMotion throughput, not by the patch itself. Designs routinely assume a window that the arithmetic does not support.
Per host evacuation:
VMs per host V
Average active memory M GB (active, not configured)
vMotion bandwidth B Gb/s usable, not link speed
Concurrent vMotions C per host, bounded by vCenter and
by the vMotion network provisioned
seconds_per_vm ~= (M x 8) / B plus stun and switchover
host_seconds ~= V / C x seconds_per_vm
cluster_seconds ~= host_seconds x hosts_to_patch
Worked: 16-host cluster, 60 VMs per host, 6 GB active average,
10 Gb/s vMotion network, 4 concurrent
per VM = (6 x 8) / 10 ~= 4.8 s of transfer
call it 8 s with overhead
per host = 60 / 4 x 8 ~= 120 s of pure migration
plus maintenance mode entry, patch, reboot, exit: 15 to 25 min
cluster = 16 x ~20 min ~= 5.3 hours, serialised
The migration is NOT the dominant term. Reboot and readiness is.
That is why designs that optimise the vMotion network and ignore
boot time miss the target.
Design lever: patch two hosts concurrently and the window halves,
but only if HA reserve tolerates M = 2. Which takes you back to
HOSTS = W + F + M, with M = 2 this time. The maintenance strategy
and the host count are the same decision.That last point is the one worth carrying into a design review. Deciding to patch two hosts at a time is not an operational preference, it is a capacity requirement, and if it was not sized for then the cluster will refuse to enter maintenance mode on the second host and the window will overrun.
05. A scale probe for your own estate
This measures the things that bind, rather than reporting the things that are documented. Run it monthly and keep the output, because the trend is more useful than any single reading.
#!/usr/bin/env python3
"""vsphere_scale_probe.py
Measures the practical ceilings of a vCenter rather than its
configuration maximums:
* inventory size by type
* Property Collector retrieval latency, ms per object
* task queue: running and recently failed
* open sessions, and which identities hold them
* per-cluster evacuation estimate
Read only. Uses PropertyCollector throughout, so it does not
itself become the load it is measuring.
"""
import argparse, ssl, sys, time
from collections import Counter
from datetime import datetime, timedelta, timezone
from pyVim.connect import SmartConnect, Disconnect
from pyVmomi import vim
from vsphere_bulk import bulk_properties
def inventory(si):
out = {}
for t, label in ((vim.VirtualMachine, "VMs"),
(vim.HostSystem, "Hosts"),
(vim.ClusterComputeResource, "Clusters"),
(vim.Datastore, "Datastores"),
(vim.Network, "Networks")):
t0 = time.perf_counter()
rows, pages = bulk_properties(si, t, ["name"])
dt = time.perf_counter() - t0
out[label] = {
"count": len(rows),
"seconds": round(dt, 2),
"ms_per_object": round(dt * 1000 / max(len(rows), 1), 2),
"pages": pages,
}
return out
def sessions(si):
sm = si.content.sessionManager
try:
live = sm.sessionList or []
except vim.fault.NoPermission:
return None
by_user = Counter(s.userName for s in live)
now = datetime.now(timezone.utc)
idle = [s for s in live
if s.lastActiveTime and (now - s.lastActiveTime) > timedelta(hours=2)]
return {"total": len(live), "by_user": by_user.most_common(10),
"idle_over_2h": len(idle)}
def tasks(si, hours=24):
tm = si.content.taskManager
running = [t for t in (tm.recentTask or [])
if t.info.state == vim.TaskInfo.State.running]
queued = [t for t in (tm.recentTask or [])
if t.info.state == vim.TaskInfo.State.queued]
failed = [t for t in (tm.recentTask or [])
if t.info.state == vim.TaskInfo.State.error]
return {"running": len(running), "queued": len(queued),
"failed_recent": len(failed),
"failed_names": [t.info.descriptionId for t in failed[:8]]}
def evacuation(si, concurrent=4, vmotion_gbps=10.0):
"""Rough per-host evacuation estimate from ACTIVE memory."""
vms, _ = bulk_properties(si, vim.VirtualMachine, [
"name", "runtime.host", "runtime.powerState",
"summary.quickStats.guestMemoryUsage"])
hosts, _ = bulk_properties(si, vim.HostSystem, ["name", "parent"])
per_host = {}
for moref, row in vms.items():
if row.get("runtime.powerState") != "poweredOn":
continue
h = row.get("runtime.host")
if not h:
continue
active_mb = row.get("summary.quickStats.guestMemoryUsage") or 0
e = per_host.setdefault(h, {"vms": 0, "active_gb": 0.0})
e["vms"] += 1
e["active_gb"] += active_mb / 1024.0
rows = []
for h, e in per_host.items():
name = hosts.get(h, {}).get("name", str(h))
if e["vms"] == 0:
continue
avg_gb = e["active_gb"] / e["vms"]
sec_per_vm = (avg_gb * 8) / vmotion_gbps * 1.6 # overhead factor
host_sec = (e["vms"] / concurrent) * sec_per_vm
rows.append((name, e["vms"], round(e["active_gb"], 1),
round(host_sec / 60, 1)))
rows.sort(key=lambda r: r[3], reverse=True)
return rows
def main():
p = argparse.ArgumentParser()
p.add_argument("--vcenter", required=True)
p.add_argument("--user", required=True)
p.add_argument("--password", required=True)
p.add_argument("--concurrent-vmotion", type=int, default=4)
p.add_argument("--vmotion-gbps", type=float, default=10.0)
a = p.parse_args()
ctx = ssl._create_unverified_context()
si = SmartConnect(host=a.vcenter, user=a.user, pwd=a.password,
sslContext=ctx)
try:
print("=== INVENTORY AND RETRIEVAL LATENCY")
for label, m in inventory(si).items():
print(f" {label:<12} {m['count']:6d} "
f"{m['seconds']:6.2f}s {m['ms_per_object']:6.2f} ms/obj "
f"{m['pages']} pages")
print("\n=== TASKS")
t = tasks(si)
print(f" running {t['running']}, queued {t['queued']}, "
f"recently failed {t['failed_recent']}")
for n in t["failed_names"]:
print(f" failed: {n}")
if t["queued"] > 0:
print(" NOTE queued tasks mean vCenter is already saturated")
print("\n=== SESSIONS")
s = sessions(si)
if s is None:
print(" insufficient permission to enumerate sessions")
else:
print(f" total {s['total']}, idle over 2h {s['idle_over_2h']}")
for user, n in s["by_user"]:
flag = " <-- investigate" if n > 10 else ""
print(f" {user:<44} {n:4d}{flag}")
if s["idle_over_2h"] > 20:
print(" NOTE many idle sessions: a script is not calling logout")
print("\n=== EVACUATION ESTIMATE (migration only, excludes reboot)")
rows = evacuation(si, a.concurrent_vmotion, a.vmotion_gbps)
print(f" {'HOST':<38}{'VMs':>5}{'ACTIVE GB':>11}{'EST MIN':>9}")
for name, vms, gb, mins in rows[:15]:
print(f" {name[:37]:<38}{vms:>5}{gb:>11.1f}{mins:>9.1f}")
if rows:
worst = rows[0]
print(f"\n worst host: {worst[0]} at ~{worst[3]} min of migration")
print(f" add 15 to 25 min per host for patch, reboot and readiness")
finally:
Disconnect(si)
if __name__ == "__main__":
sys.exit(main())06. What the numbers should prompt
| Observation | What it means | Design response |
|---|---|---|
| ms per object rising month on month at constant inventory | vCenter is degrading under accumulating load, often database growth or a new integration | Check task and event retention. Audit what was added since the last reading |
| Tasks in queued state at all | Already saturated. Every further request is making it worse | Split the estate across vCenters, or find the client generating the volume |
| One identity holding dozens of sessions | A script that never logs out | Fix the script. Give each integration its own account so the next one is attributable |
| Evacuation estimate exceeding the window | The maintenance plan is arithmetically impossible | Increase concurrency, which requires HA headroom, which changes host count |
| One host far above the others | DRS imbalance, or affinity rules concentrating memory | Review rules before assuming DRS is misconfigured |
The value here is the trend, not the absolute figure. A retrieval latency of 4 ms per object means nothing on its own. The same estate reading 4 ms in January and 11 ms in June means something has been added, and finding it while it is still a graph is considerably cheaper than finding it during an incident.
Property Collector semantics are stable across vSphere releases, but pyVmomi type paths, session enumeration permissions and quickStats field availability differ by version, so confirm against your build. The benchmark deliberately generates load on the naive path; run it out of hours and start with a small limit. Nothing here is official guidance from VMware, Broadcom or any vendor.
Leave a Reply