Part 6 of 10. Nobody deletes snapshots. They are taken before a change, the change goes fine, and the snapshot stays for two years holding capacity and quietly changing the write path.

Why this matters more on a per-VM array

  • Capacity is held, not freed. Every block the snapshot references stays allocated even after the live VM has moved past it.
  • Long chains change the performance profile. This is why an old snapshot shows up as a latency complaint months after anyone remembers taking it.
  • Array snapshots and vSphere snapshots are different things. A VM can be clean in the vSphere Client and still carry a dozen array-side snapshots. Audit both.

Distinguish scheduled from manual before you delete anything. Snapshots created by a protection policy are supposed to exist and will be aged out by that policy. The ones worth chasing are manual snapshots with no expiry, taken by a person, during a change window that closed long ago. Both scripts below separate them.

PowerShell

#requires -Version 5.1
<#
  tintri-snapshot-audit.ps1
  Ranks VMs by oldest snapshot and reports what is reclaimable.
  Reports only. Deletion is deliberately left to a human.
#>

param(
    [Parameter(Mandatory)][string]$VMstore,
    [Parameter(Mandatory)][pscredential]$Credential,
    [int]$AgeWarnDays = 7,
    [int]$AgeCritDays = 30,
    [string]$CsvPath
)

Import-Module TintriPSToolkit -ErrorAction Stop
$conn = Connect-TintriServer -Server $VMstore -Credential $Credential -SetDefaultServer

try {
    $now  = Get-Date
    $rows = foreach ($vm in Get-TintriVM) {

        $snaps = Get-TintriVMSnapshot -VM $vm -ErrorAction SilentlyContinue
        if (-not $snaps) { continue }

        $oldest = ($snaps | Sort-Object CreateTime | Select-Object -First 1)
        $ageDays = [math]::Round(($now - $oldest.CreateTime).TotalDays, 1)

        # scheduled snapshots carry a policy type; manual ones do not
        $manual = @($snaps | Where-Object { $_.Type -ne 'SCHEDULED' })

        [pscustomobject]@{
            VM            = $vm.VmwareVM.Name
            SnapCount     = $snaps.Count
            ManualCount   = $manual.Count
            OldestDays    = $ageDays
            OldestName    = $oldest.Description
            Severity      = if     ($ageDays -ge $AgeCritDays) { 'CRITICAL' }
                            elseif ($ageDays -ge $AgeWarnDays) { 'WARN' }
                            else                               { 'ok' }
        }
    }

    $rows |
        Sort-Object OldestDays -Descending |
        Format-Table VM, SnapCount, ManualCount, OldestDays, Severity -AutoSize

    $crit = @($rows | Where-Object Severity -eq 'CRITICAL')
    $warn = @($rows | Where-Object Severity -eq 'WARN')

    Write-Host "`nSUMMARY" -ForegroundColor Cyan
    "  VMs with snapshots        : {0}" -f $rows.Count
    "  Older than $AgeCritDays days       : {0}" -f $crit.Count
    "  Older than $AgeWarnDays days        : {0}" -f $warn.Count
    "  Total manual snapshots    : {0}" -f (($rows | Measure-Object ManualCount -Sum).Sum)

    if ($crit) {
        Write-Host "`nRECLAIM CANDIDATES (manual, over $AgeCritDays days)" -ForegroundColor Yellow
        $crit | Where-Object ManualCount -gt 0 | ForEach-Object {
            "  {0,-38} {1} manual, oldest {2} days" -f `
                $_.VM, $_.ManualCount, $_.OldestDays
        }
    }

    if ($CsvPath) {
        $rows | Sort-Object OldestDays -Descending |
            Export-Csv -Path $CsvPath -NoTypeInformation
        Write-Host "`nWritten to $CsvPath" -ForegroundColor Green
    }
}
finally {
    if ($conn) { Disconnect-TintriServer -TintriServer $conn }
}

Python

#!/usr/bin/env python3
"""tintri_snapshot_audit.py - snapshot sprawl report from a VMstore.

Report only. Nothing here deletes anything.
"""

import argparse
import csv
import os
import sys
from datetime import datetime, timezone

import urllib3
import requests

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
CREDS_TYPE = "com.tintri.api.rest.vcommon.dto.rbac.RestApiCredentials"


def parse_ts(value):
    """VMstore timestamps are ISO-ish; be forgiving about the format."""
    if value is None:
        return None
    if isinstance(value, (int, float)):
        return datetime.fromtimestamp(value / 1000, tz=timezone.utc)
    txt = str(value).replace("Z", "+00:00")
    try:
        dt = datetime.fromisoformat(txt)
        return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
    except ValueError:
        return None


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--vmstore", required=True)
    p.add_argument("--user", default=os.environ.get("TINTRI_USER"))
    p.add_argument("--password", default=os.environ.get("TINTRI_PASS"))
    p.add_argument("--api-version", default="v310")
    p.add_argument("--warn-days", type=int, default=7)
    p.add_argument("--crit-days", type=int, default=30)
    p.add_argument("--csv")
    a = p.parse_args()

    if not a.user or not a.password:
        sys.exit("set TINTRI_USER and TINTRI_PASS")

    base = f"https://{a.vmstore}/api/{a.api_version}"
    s = requests.Session()
    s.verify = False
    r = s.post(f"{base}/session/login",
               json={"typeId": CREDS_TYPE,
                     "username": a.user, "password": a.password},
               timeout=30)
    if r.status_code == 404:
        sys.exit(f"404 on {base}. Try a fully qualified version such as "
                 "v310.203, or check GET /api/info.")
    r.raise_for_status()

    now = datetime.now(timezone.utc)
    rows = []

    try:
        vms = s.get(f"{base}/vm", timeout=60).json().get("items", [])
        for vm in vms:
            uuid = (vm.get("uuid") or {}).get("uuid")
            name = (vm.get("vmware") or {}).get("name", "unknown")
            if not uuid:
                continue

            resp = s.get(f"{base}/vm/{uuid}/snapshot", timeout=60)
            if resp.status_code != 200:
                continue
            snaps = resp.json().get("items", [])
            if not snaps:
                continue

            times = [parse_ts(sn.get("createTime")) for sn in snaps]
            times = [t for t in times if t]
            if not times:
                continue

            age = round((now - min(times)).total_seconds() / 86400, 1)
            manual = sum(1 for sn in snaps
                         if str(sn.get("type", "")).upper() != "SCHEDULED")

            severity = ("CRITICAL" if age >= a.crit_days
                        else "WARN" if age >= a.warn_days
                        else "ok")

            rows.append({"vm": name, "count": len(snaps), "manual": manual,
                         "oldest_days": age, "severity": severity})
    finally:
        s.get(f"{base}/session/logout", timeout=15)

    rows.sort(key=lambda r: r["oldest_days"], reverse=True)

    hdr = f"{'VM':<38}{'SNAPS':>7}{'MANUAL':>8}{'OLDEST(d)':>11}   SEVERITY"
    print(hdr)
    print("-" * (len(hdr) + 2))
    for r in rows:
        print(f"{r['vm'][:37]:<38}{r['count']:>7}{r['manual']:>8}"
              f"{r['oldest_days']:>11.1f}   {r['severity']}")

    crit = [r for r in rows if r["severity"] == "CRITICAL"]
    print(f"\nVMs with snapshots: {len(rows)}")
    print(f"Older than {a.crit_days} days: {len(crit)}")
    print(f"Total manual snapshots: {sum(r['manual'] for r in rows)}")

    if crit:
        print(f"\nReclaim candidates (manual, over {a.crit_days} days):")
        for r in crit:
            if r["manual"]:
                print(f"  {r['vm'][:37]:<38} {r['manual']} manual, "
                      f"oldest {r['oldest_days']} days")

    if a.csv:
        with open(a.csv, "w", newline="") as fh:
            w = csv.DictWriter(fh, fieldnames=list(rows[0].keys()) if rows else
                               ["vm", "count", "manual", "oldest_days", "severity"])
            w.writeheader()
            w.writerows(rows)
        print(f"\nWritten to {a.csv}")

    return 0 if not crit else 2


if __name__ == "__main__":
    sys.exit(main())

Schedule it weekly and mail the CSV. Neither script deletes anything, which is deliberate. An automated snapshot deletion is a bad idea on any array, and a very bad idea on one where a snapshot might be the only copy of something.

Next: auditing per-VM QoS, and finding the ceiling somebody set during an incident and never removed.

Snapshot object field names, particularly the type and timestamp keys, vary by TxOS build. Dump one snapshot object and confirm before scheduling. A read only account is sufficient for the audit.

Leave a Reply

Discover more from VMwareBlogs

Subscribe now to keep reading and get access to the full archive.

Continue reading