Part 8 of 10. Certificate rotation in VCF is not a certificate problem. It is a distributed state problem wearing a certificate costume. SDDC Manager holds one view of the trust chain, each component holds its own, and when a rotation stops halfway those views disagree in a way that no single console will show you.
This post is about the failure, not the happy path. The happy path is a wizard. What follows is what to do when the wizard stops at 60 percent and the SDDC Manager UI will no longer load.
01. Why this is harder than replacing a web server certificate
A normal certificate replacement has one holder of truth: the server. VCF has at least four, and they validate each other.
+---------------------------+
| SDDC Manager |
| believes it knows every |
| component's certificate |
+------------+--------------+
validates | validates
+-------------+----------+---------+-------------+
v v v v
+---------+ +-----------+ +-----------+ +----------+
| vCenter | | NSX | | VCF Ops | | ESXi |
| VMCA is | | node cert | | | | signed |
| its own | | + | | | | by VMCA |
| CA | | VIP cert | | | | |
+----+----+ +-----------+ +-----------+ +----------+
|
+--> vCenter's VMCA signs ESXi host certificates, so a
vCenter certificate change can invalidate every host
Four independent trust stores. A rotation must update all of them,
in order, and SDDC Manager must be told the truth afterwards.The single most damaging mistake is replacing a vCenter or NSX certificate outside VCF. Doing it directly in the vSphere Client or the NSX UI succeeds locally and leaves SDDC Manager holding a thumbprint that no longer exists. Nothing errors at the time. It errors weeks later, during an upgrade precheck, and the diagnosis points nowhere near certificates.
02. Detect drift before you rotate anything
The audit that matters is not “when does this expire”. It is “does what the component actually serves on the wire match what SDDC Manager believes it serves”. Those are different questions and only the second one predicts an outage.
The script below asks four things per component: what is served on port 443, what SDDC Manager reports, whether the SANs cover every name in use, and whether the chain validates against the trust store.
#!/usr/bin/env python3
"""vcf_cert_drift.py
Compares the certificate each VCF component actually serves against what
SDDC Manager believes it has. Reports expiry, SAN coverage, chain validity
and thumbprint drift.
Drift is the finding that matters. Expiry you can plan for; drift is a
failed rotation that nobody noticed.
"""
import argparse
import hashlib
import socket
import ssl
import sys
from datetime import datetime, timezone
import urllib3
import requests
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
try:
from cryptography import x509
from cryptography.hazmat.backends import default_backend
except ImportError:
sys.exit("pip install cryptography")
def fetch_leaf(host, port=443, timeout=10):
"""Pull the served leaf certificate without validating it.
Deliberately unvalidated: we want to inspect a certificate that may
already be broken. Validation is a separate check below.
"""
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
with socket.create_connection((host, port), timeout=timeout) as sock:
with ctx.wrap_socket(sock, server_hostname=host) as tls:
der = tls.getpeercert(binary_form=True)
return x509.load_der_x509_certificate(der, default_backend()), der
def chain_validates(host, port=443, timeout=10):
"""Does the served chain validate against the system trust store?"""
ctx = ssl.create_default_context()
try:
with socket.create_connection((host, port), timeout=timeout) as sock:
with ctx.wrap_socket(sock, server_hostname=host):
return True, ""
except ssl.SSLCertVerificationError as e:
return False, e.verify_message or str(e)
except Exception as e:
return False, str(e)
def sans(cert):
try:
ext = cert.extensions.get_extension_for_class(
x509.SubjectAlternativeName)
return set(ext.value.get_values_for_type(x509.DNSName)) | {
str(ip) for ip in ext.value.get_values_for_type(x509.IPAddress)}
except x509.ExtensionNotFound:
return set()
def sha256_thumb(der):
return hashlib.sha256(der).hexdigest().upper()
class Sddc:
def __init__(self, host, user, pwd):
self.base = f"https://{host}/v1"
self.s = requests.Session()
self.s.verify = False
r = self.s.post(f"{self.base}/tokens",
json={"username": user, "password": pwd}, timeout=30)
r.raise_for_status()
self.s.headers["Authorization"] = f"Bearer {r.json()['accessToken']}"
def get(self, path):
r = self.s.get(f"{self.base}{path}", timeout=60)
r.raise_for_status()
return r.json()
def domain_certificates(self, domain_id):
"""What SDDC Manager believes each resource is presenting."""
try:
return self.get(f"/domains/{domain_id}/resource-certificates") \
.get("elements", [])
except requests.HTTPError:
return []
def main():
p = argparse.ArgumentParser()
p.add_argument("--sddc", required=True)
p.add_argument("--user", required=True)
p.add_argument("--password", required=True)
p.add_argument("--warn-days", type=int, default=60)
a = p.parse_args()
sddc = Sddc(a.sddc, a.user, a.password)
now = datetime.now(timezone.utc)
# build the list of endpoints VCF cares about
targets = [("sddc-manager", a.sddc, None)]
believed = {}
for d in sddc.get("/domains").get("elements", []):
for rec in sddc.domain_certificates(d["id"]):
fqdn = rec.get("resourceFqdn") or rec.get("fqdn")
if not fqdn:
continue
targets.append((rec.get("resourceType", "?"), fqdn, d.get("name")))
believed[fqdn.lower()] = {
"thumb": (rec.get("thumbprint")
or rec.get("certificateThumbprint") or "").upper()
.replace(":", ""),
"expiry": rec.get("notAfter") or rec.get("expirationDate"),
"issuer": rec.get("issuedBy") or rec.get("issuer"),
}
print(f"{'COMPONENT':<14}{'FQDN':<34}{'DAYS':>6} {'CHAIN':<7}"
f"{'SAN':<5}{'DRIFT':<7} ISSUER")
print("-" * 108)
problems = []
for kind, fqdn, domain in targets:
try:
cert, der = fetch_leaf(fqdn)
except Exception as e:
print(f"{kind:<14}{fqdn:<34}{'--':>6} UNREACHABLE ({e})")
problems.append((fqdn, "unreachable", str(e)))
continue
not_after = cert.not_valid_after_utc if hasattr(
cert, "not_valid_after_utc") else cert.not_valid_after.replace(
tzinfo=timezone.utc)
days = (not_after - now).days
ok_chain, why = chain_validates(fqdn)
names = sans(cert)
san_ok = fqdn.lower() in {n.lower() for n in names}
served_thumb = sha256_thumb(der)
rec = believed.get(fqdn.lower())
if rec and rec["thumb"]:
drift = "DRIFT" if rec["thumb"] != served_thumb else "ok"
else:
drift = "n/a"
issuer = ""
try:
issuer = cert.issuer.rfc4514_string()[:34]
except Exception:
pass
print(f"{kind:<14}{fqdn:<34}{days:>6} "
f"{'ok' if ok_chain else 'BAD':<7}"
f"{'ok' if san_ok else 'NO':<5}{drift:<7} {issuer}")
if days < a.warn_days:
problems.append((fqdn, "expiring", f"{days} days"))
if not ok_chain:
problems.append((fqdn, "chain", why))
if not san_ok:
problems.append((fqdn, "san",
f"{fqdn} not in SAN list {sorted(names)}"))
if drift == "DRIFT":
problems.append((fqdn, "drift",
"served certificate differs from the one "
"SDDC Manager has recorded"))
if problems:
print("\nFINDINGS")
for fqdn, kind, detail in problems:
print(f" [{kind.upper():<11}] {fqdn}\n {detail}")
drifted = [p for p in problems if p[1] == "drift"]
if drifted:
print("\nDRIFT IS THE URGENT ONE. SDDC Manager is holding a stale")
print("thumbprint. Upgrades and workflows that validate trust will")
print("fail with errors that do not mention certificates. See the")
print("recovery section before attempting any rotation.")
return 0 if not problems else 2
if __name__ == "__main__":
sys.exit(main())03. The same check from PowerShell
#requires -Version 7.0
<#
vcf-cert-drift.ps1
Served certificate versus what SDDC Manager believes.
#>
param(
[Parameter(Mandatory)][string]$SddcManager,
[Parameter(Mandatory)][string]$Username,
[Parameter(Mandatory)][securestring]$Password,
[int]$WarnDays = 60
)
$base = "https://$SddcManager/v1"
$skip = @{ SkipCertificateCheck = $true }
$plain = [System.Net.NetworkCredential]::new('', $Password).Password
$tok = (Invoke-RestMethod -Method Post -Uri "$base/tokens" `
-Body (@{username=$Username;password=$plain} | ConvertTo-Json) `
-ContentType 'application/json' @skip).accessToken
$hdr = @{ Authorization = "Bearer $tok" }
function Get-ServedCert {
param([string]$Fqdn, [int]$Port = 443)
$client = [System.Net.Sockets.TcpClient]::new()
try {
$client.Connect($Fqdn, $Port)
# accept anything: we are inspecting, not trusting
$ssl = [System.Net.Security.SslStream]::new(
$client.GetStream(), $false, { $true })
$ssl.AuthenticateAsClient($Fqdn)
$raw = $ssl.RemoteCertificate
$cert = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($raw)
# SHA256 thumbprint, since VCF reports SHA256 in recent releases
$sha = [System.Security.Cryptography.SHA256]::Create()
$thumb = ($sha.ComputeHash($cert.RawData) |
ForEach-Object { $_.ToString('X2') }) -join ''
$sanExt = $cert.Extensions |
Where-Object { $_.Oid.Value -eq '2.5.29.17' }
$san = if ($sanExt) { $sanExt.Format($false) } else { '' }
[pscustomobject]@{
Subject = $cert.Subject
Issuer = $cert.Issuer
NotAfter = $cert.NotAfter
DaysLeft = [int]($cert.NotAfter - (Get-Date)).TotalDays
Thumb256 = $thumb
San = $san
SanCovers = $san -match [regex]::Escape($Fqdn)
}
}
finally { $client.Dispose() }
}
$rows = foreach ($d in (Invoke-RestMethod "$base/domains" -Headers $hdr @skip).elements) {
$certs = try {
(Invoke-RestMethod "$base/domains/$($d.id)/resource-certificates" `
-Headers $hdr @skip).elements
} catch { @() }
foreach ($c in $certs) {
$fqdn = if ($c.resourceFqdn) { $c.resourceFqdn } else { $c.fqdn }
if (-not $fqdn) { continue }
try { $served = Get-ServedCert -Fqdn $fqdn }
catch {
[pscustomobject]@{ Domain=$d.name; Fqdn=$fqdn; Days='--'
SanOk='--'; Drift='UNREACHABLE' }
continue
}
$believed = ($c.thumbprint, $c.certificateThumbprint |
Where-Object { $_ })[0]
$believed = if ($believed) { $believed.ToUpper().Replace(':','') } else { '' }
[pscustomobject]@{
Domain = $d.name
Fqdn = $fqdn
Days = $served.DaysLeft
SanOk = $served.SanCovers
Drift = if (-not $believed) { 'n/a' }
elseif ($believed -ne $served.Thumb256) { 'DRIFT' }
else { 'ok' }
Issuer = $served.Issuer
}
}
}
$rows | Format-Table -AutoSize
$drift = @($rows | Where-Object Drift -eq 'DRIFT')
$soon = @($rows | Where-Object { $_.Days -is [int] -and $_.Days -lt $WarnDays })
if ($drift) {
Write-Host "`n$($drift.Count) component(s) have drifted from SDDC Manager's record." -ForegroundColor Red
Write-Host 'Fix drift before attempting any rotation or upgrade.' -ForegroundColor Yellow
}
if ($soon) {
Write-Host "`n$($soon.Count) certificate(s) expire within $WarnDays days." -ForegroundColor Yellow
}04. The four ways rotation strands, and what each looks like
Failure 1: replaced outside VCF, SDDC Manager never told
Symptom. Everything works. Weeks later an upgrade precheck fails, or a workflow errors with a connectivity or SSL peer verification message that names no certificate. The drift column in the script above reads DRIFT.
Mechanism. The component serves a new certificate. SDDC Manager’s inventory still holds the old thumbprint. Any workflow that validates trust before acting compares the two, finds a mismatch, and aborts with whatever error that workflow raises, which is rarely a certificate error.
Recovery. Do not rotate again. Re-trigger SDDC Manager’s discovery of that resource so it re-reads the served certificate, then re-run the drift script and confirm the column reads ok. If discovery does not clear it, the trust store on SDDC Manager is missing the new issuing CA, which is failure 3.
Failure 2: rotation stops partway through a multi-component domain
Symptom. The task shows failed. Some components now present the new certificate, others still present the old one. The SDDC Manager UI may itself be unreachable if its own certificate was in the batch.
Mechanism. Rotation is a sequence of per-resource operations, not a transaction. There is no rollback. Whatever completed stays completed.
# establish ground truth before touching anything
# what does each component actually serve, right now
for h in sddc-mgr vcenter-mgmt nsx-mgmt-a nsx-mgmt-b nsx-vip; do
echo "=== $h"
echo | openssl s_client -connect ${h}.lab.local:443 -servername ${h}.lab.local 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates -fingerprint -sha256
done
# then the failed task and its subtasks, which name the exact resource
curl -sk -H "Authorization: Bearer $TOKEN" \
https://sddc-mgr.lab.local/v1/tasks?status=Failed | python3 -m json.toolRecovery. Identify the exact resource the task stopped on, resolve that one cause, then re-run the rotation scoped to the remaining resources only. Re-running the whole batch against components that already succeeded is how a partial failure becomes a total one.
Failure 3: the issuing CA is not in the trust store
Symptom. Rotation fails immediately at validation with a chain or untrusted issuer message. The chain column in the script reads BAD.
Mechanism. Usually an intermediate CA was omitted when the signed certificate was returned. The leaf is valid, the root is trusted, and the link between them is missing. A browser may hide this by fetching the intermediate itself. VCF will not.
# count the certificates the server actually sends
echo | openssl s_client -connect vcenter.lab.local:443 -showcerts 2>/dev/null \
| grep -c 'BEGIN CERTIFICATE'
# 1 means leaf only. You need the intermediates too.
# verify the chain explicitly against your root
openssl verify -CAfile root-ca.pem -untrusted intermediate.pem leaf.pem
# build the correct order: leaf first, then intermediates, root last
cat leaf.pem intermediate.pem root-ca.pem > fullchain.pemRecovery. Rebuild the file in the correct order, leaf first, and re-submit. Order matters and is a common cause of a file that looks complete failing validation.
Failure 4: hosts disconnect after a vCenter certificate change
Symptom. vCenter comes back healthy and then ESXi hosts show as not responding. VMs keep running, because this is a management plane failure, not a data plane one.
Mechanism. vCenter’s VMCA signs host certificates. Changing vCenter’s machine certificate, or worse changing VMCA’s own signing certificate, breaks the trust relationship every host relies on. This is the failure that turns a routine change into an incident, and it is why the vCenter step is the one to schedule a window for.
Recovery. Reconnect the affected hosts so they re-establish trust and, where required, renew their certificates from the new VMCA. Do this host by host and confirm each returns to connected before moving on. Reconnecting an entire cluster at once during a trust failure produces a stampede of tasks that is very hard to read.
05. Order of operations, and why it is this order
| Step | Component | Why here | Blast radius if it fails |
|---|---|---|---|
| 1 | Trust store on SDDC Manager | Nothing validates without the CA present first | Rotation refuses to start, nothing changes |
| 2 | SDDC Manager itself | Prove the process on the component you can most easily recover | UI unreachable, API may still answer |
| 3 | NSX Manager nodes, then the VIP | The VIP certificate is separate and is routinely forgotten | NSX UI or API partially unreachable |
| 4 | vCenter | Last, because it can take every ESXi host with it | Hosts disconnect, management plane outage |
| 5 | Re-run the drift audit | Confirm SDDC Manager’s record matches reality | Silent drift, fails weeks later |
Step 5 is not optional and it is the one people skip. A rotation that reports success can still leave SDDC Manager holding a stale thumbprint for one resource. That is failure 1, arriving on a delay. Run the drift script immediately after every rotation and keep the output with the change record.
06. Before you start
- Take a snapshot or backup of SDDC Manager and vCenter. Certificate state is not something you can hand-edit back into consistency.
- Confirm every SAN before submitting. The certificate must cover the FQDN, the short name and the IP if anything connects by IP. A missing SAN passes generation and fails at use.
- Check clock skew across all appliances. A certificate whose validity window has not opened yet, because one appliance believes it is yesterday, fails with an expiry error that makes no sense against the dates you are reading.
- Have console access to every appliance. If you lose the management interface partway through, SSH or the VM console is your only route back.
- Do it in a maintenance window even though nothing should go down. Failure 4 is a management plane outage and it does not announce itself in advance.
Next in the series: pre-upgrade prechecks, which is the other place stale trust state surfaces, usually as an error that names something entirely unrelated.
Endpoint paths and field names for resource certificates vary across VCF releases, and some releases report SHA1 thumbprints rather than SHA256. Confirm both against your build before trusting the drift comparison, since a hash algorithm mismatch will report drift on every component. Nothing here is official guidance.
Leave a Reply