Category: VMware

ESXi is the hypervisor in a vSphere environment. The hypervisor provides physical hardware resources dynamically to virtual machines as needed to support the operation of the virtual machines. The hypervisor allows virtual machines to operate with a degree of independence from the underlying physical hardware. For example, a virtual machine can be moved from one physical host to another, or its virtual disks can be moved from one type of storage to another, without affecting the functioning of the virtual machine.
Because virtual machines are decoupled from specific underlying physical hardware, virtualization allows you to consolidate physical computing resources such as CPUs, memory, storage, and networking into pools of resources that can be dynamically and flexibly made available to virtual machines. With appropriate management software, such as vCenter Server, you can also use a number of features that increase the availability and security of your virtual infrastructure.

Power Shell scripts for Virtual machines

tapasmahanta1241 Comment

List of each of your vms:

Get-VM | Where {$_.PowerState -eq "PoweredOn"} | 
Select Name, Host, NumCpu, MemoryMB, <code>
 @{N=&quot;Cpu.UsageMhz.Average&quot;;E={[Math]::Round((($_ |Get-Stat -Stat cpu.usagemhz.average -Start (Get-Date).AddHours(-24)-IntervalMins 5 -MaxSamples (12) |Measure-Object Value -Average).Average),2)}}, </code>
 @{N=&quot;Mem.Usage.Average&quot;;E={[Math]::Round((($_ |Get-Stat -Stat mem.usage.average -Start (Get-Date).AddHours(-24)-IntervalMins 5 -MaxSamples (12) |Measure-Object Value -Average).Average),2)}} `
 | Export-Csv c:\Temp\Temp.csv

VMs per Resource Pool:

Get-ResourcePool | Where {$_.Name -ne “Resources“ }| Select Name, @{N=“NumVM“;E={($_ | Get-VM).Count}} | Sort Name|Export-Csv -NoTypeInformation c:\Temp.csv

 

SVMotion via PowerCLI :

Get-VM “Test“ |Move-VM -datastore (Get-datastore “MyDatastore“)

 

The Move-VM Cmdlet covers a multiple of sins, lets check some out, you want VMotion:

Get-VM -Name “Test“ |Move-VM -Destination (Get-VMHost MyHost)

 

Moving a VM to a new folder:

Move-VM -VM (Get-VM -Name MyVM)-Destination (Get-Folder -Name Production)

 

Moving a VM to a new resource pool:

Move-VM -VM (Get-VM -Name MyVM)-Destination (Get-ResourcePool -Name “Important“)

VM guest Disk size:



ForEach ($VM in Get-VM ){($VM.Extensiondata.Guest.Disk | Select @{N="Name";E={$VM.Name}},DiskPath, @{N="Capacity(MB)";E={[math]::Round($_.Capacity/ 1MB)}}, @{N="Free Space(MB)";E={[math]::Round($_.FreeSpace / 1MB)}}, @{N="Free Space %";E={[math]::Round(((100* ($_.FreeSpace))/ ($_.Capacity)),0)}})}

PowerShell Scripts for Esxi

tapasmahanta1241 Comment

List all ESX Hosts and their Service console information:

Get-VMHost | Get-VMHostNetwork | Select Hostname, ConsoleGateway, DNSAddress -ExpandProperty ConsoleNic | Select Hostname, PortGroupName, IP, SubnetMask, ConsoleGateway, DNSAddress, Devicename

 

Processor types:

Get-VMHost | Sort Name | Get-View | Select Name, @{N=“Cluster“;E={Get-Cluster -VMHost (Get-VMHost $_.Name)}},@{N=“CPU“;E={$_.Hardware.CpuPkg[0].Description}} | Export-Csv c:\Test.csv

 

NTP:

Get-VMHost |Sort Name|Select Name, @{N=“NTPServer“;E={$_ |Get-VMHostNtpServer}}, @{N=“ServiceRunning“;E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq “ntpd“}).Running}}

 

Host Hardware :

Get-VMHost |Sort Name |Get-View |
Select Name, 
@{N=“Type“;E={$_.Hardware.SystemInfo.Vendor+ “ “ + $_.Hardware.SystemInfo.Model}},
@{N=“CPU“;E={“PROC:“ + $_.Hardware.CpuInfo.NumCpuPackages + “ CORES:“ + $_.Hardware.CpuInfo.NumCpuCores + “ MHZ: “ + [math]::round($_.Hardware.CpuInfo.Hz / 1000000, 0)}},
@{N=“MEM“;E={“” + [math]::round($_.Hardware.MemorySize / 1GB, 0) + “ GB“}} | Export-Csv c:\Test.csv

 

VMs per Resource Pool:

Get-ResourcePool | Where {$_.Name -ne “Resources“ }| Select Name, @{N=“NumVM“;E={($_ | Get-VM).Count}} | Sort Name|Export-Csv -NoTypeInformation c:\Test.csv

 

Check VM nic speed:

Connect-VIServer MYVISERVER
 
$VMhosts = Get-VMHost | Get-View
Foreach ($vmhost in $vmhosts){
 Write-Output $vmhost.Name
 $pnic = 0
 Do { 
 $Speed = $VMhost.Config.Network.Pnic[$pnic].LinkSpeed.SpeedMb 
 Write "Pnic$pnic $Speed" 
 $pnic ++
 } Until ($pnic -eq ($VMhost.Config.Network.Pnic.Length))}

 

PowerShell Scripts for Datastore

tapasmahanta1241 Comment

Number of VMs in Datastore:

Get-Datastore | Select Name, @{N="NumVM";E={@($_ | Get-VM).Count}} | Sort Name

The following script was used to add the datastore’s from a csv file :

$CSVFile = "C:\Temp\Test.csv" ## Where Test.csv is the csv file from where you want to copy from 
$MYESXHost = "Esxi.local" 
$VMHBA = "vmhba2:0:"

Connect-VIServer VCENTERSERVER ##vCenter Server 
$CSV = Import-CSV $CSVFile ## Importing CSV 
$ESXHost = Get-VMHost $MYESXHost 
$ESXHost |Get-VMHostStorage -RescanAllHBA 
Foreach ($Item in $CSV){ 
 $HostID = $Item.HostId 
 $LunID = $Item.LunID 
 Write "Creating SAN_HOST$($HostID)_LUN$($LunID)…" 
 $ESXHost |New-Datastore -Vmfs -Name "SAN_HOST$($HostID)_LUN$($LunID)" -Path "$($VMHBA)$($HostID)" 
}

List all the VMDK files for VMs stored on your local storage:

Get-Datastore |where {$_.Name -match “local“} |Get-VM |Get-HardDisk |select Filename |Export-Csv c:\Test.csv

You can also add multiple names for your local storage by added them with a | .

Get-Datastore |where {$_.Name -match “store|local|storage“} |Get-VM |Get-HardDisk |select Filename |Export-Csv c:\Test.csv

Some useful vmkfstools ‘hidden’ options

tapasmahanta1241 Comment

1. Display hosts which are actively using a volume

~ # vmkfstools –activehosts /vmfs/volumes/TEST-DATASTORE
Found 1 actively heartbeating hosts on volume ‘/vmfs/volumes/TEST-DATASTORE’
(1): MAC address 91:1a:b2:7a:14:d4

This option will show the management interface MAC address of any hosts which is actively using a datastore.

2.Display File Metadata, including Lock Information

~ # vmkfstools -D /vmfs/volumes/TEST-DATASTORE/TEST/TEST.vmdk
Lock [type 10c00001 offset 456325421v 44, hb offset 3268682
gen 6155, mode 0, owner 00000000-00000000-0000-000000000000 mtime 12783 nHld 0 n Ovf 0]
Addr <4, 558, 4>, gen 6, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 492, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 8192

3.Display VMFS File Block Details

~ # vmkfstools -P -v10 /vmfs/volumes/TEST-DATASTORE
VMFS-5.54 file system spanning 1 partitions.
File system label (if any): TEST-20
Mode: public
Capacity xxxxxxxxxxx (xxxxxx file blocks * xxxxxx), xxxxxxxxx(xxxxxxx blocks) avail
Volume Creation Time: Tue Feb 21 12:18:44 2012
Files (max/free): xxxxxx/xxxxx
Ptr Blocks (max/free): xxxxxx/xxxxxxx
Sub Blocks (max/free): xxxxxxx/xxxxxxx
Secondary Ptr Blocks (max/free): 256/256
File Blocks (overcommit/used/overcommit %): 0/115092/0
Ptr Blocks  (overcommit/used/overcommit %): 0/133/0
Sub Blocks  (overcommit/used/overcommit %): 0/6/0
UUID: xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx
Partitions spanned (on “lvm”):
naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:1
DISKLIB-LIB   : Getting VAAI support status for /vmfs/volumes/TEST-DATASTORE
Is Native Snapshot Capable: NO

4:VMDK Block Mappings

This command is very useful for displaying the mappings of VMFS file blocks to a VMDK. It can also be used to display the layout of a VMDK if you are concerned that a thin provisioned VMDK might be fragmented from a block allocation perspective.

~ # vmkfstools -t0 /vmfs/volumes/TEST-DATATSORE/TEST/TEST.vmdk

What is VMware Raw Device Mapping (RDM)?

tapasmahanta1241 Comment

In other contexts, such as the VirtualCenter client console, raw device mapping may be
described in terms such as “Mapping a VMFS volume into a datastore,” “Mapping a system LUN”
or “mapping a disk file to a physical disk volume.” These terms all refer to raw device mapping.
The following terms are used in this document or related documentation:
• Raw Disk — A disk volume accessed by a virtual machine as an alternative to a virtual disk
file; it may or may not be accessed via a mapping file.
• Raw Device — Any SCSI device accessed via a mapping file. For ESX Server 2.5, only disk
devices are supported.
• Raw LUN — A logical disk volume located in a SAN.
• LUN — Acronym for a logical unit number.
• Mapping File — A VMFS file containing metadata used to map and manage a raw device.
• Mapping — An abbreviated term for a raw device mapping.
• Mapped Device — A raw device managed by a mapping file.
• Metadata File — A mapping file.
• Compatibility Mode — The virtualization type used for SCSI device access (physical or
virtual).
• SAN — Acronym for a storage area network.
• VMFS — A high-performance file system used by VMware ESX Server

Benefits of Raw Device Mapping:

User-Friendly Persistent Names
Raw device mapping provides a user-friendly name for a mapped device — the name of its
mapping file. When you use a mapping, you don’t need to refer to the device by its device
name, as required with previous versions of ESX Server. You refer to it by the name of the
mapping file — for example, use:
/vmfs/myvmfsvolume/myrawdisk.vmdk.

Dynamic Name Resolution
Raw device mapping stores unique identification information for each mapped device. The
VMFS file system resolves each mapping to its current SCSI device, regardless of changes in the
physical configuration of the server due to adapter hardware changes, path changes, device
relocation, and so forth.

Distributed File Locking
Raw device mapping makes it possible to use VMFS distributed locking for raw SCSI devices.
Distributed locking on a raw device mapping makes it safe to use a shared raw LUN without
losing data when two virtual machines on different servers access the same LUN.

File Permissions
Raw device mapping makes file permissions possible. The permissions of the mapping file are
applied at file open time to protect the mapped volume. Previously, permissions for raw devices
could not be enforced by the file system.

File System Operations
Raw device mapping makes it possible to use file system utilities to work with a mapped
volume, using the mapping file as a proxy. Most operations that are valid for an ordinary file can
be applied to the mapping file, and are redirected to operate on the mapped device.

Redo Logs
Raw device mapping makes it possible to keep a redo log for a mapped volume. The redo log
has the name of the mapping file, with .REDO appended. Note that redo logs are not possible
when raw device mapping is used in physical compatibility mode.

VMotion
Raw device mapping allows you to migrate a virtual machine with VMotion. Previously, this was
only possible for virtual machines that used virtual disk files. When you use raw device mapping,
the mapping file acts as a proxy to allow VirtualCenter to migrate the virtual machine using the
same mechanism that exists for virtual disk files.

Note: You cannot migrate virtual machines with raw, clustered, or non-persistent mode disks
using VMotion. If you have clustered disks, you can store them on separate VMFS volumes from
the virtual machines prior to migrating them using VMotion.

SAN Management Agents
Raw device mapping makes it possible to run some SAN management agents inside a virtual
machine. Similarly, any software that needs to access a device using raw SCSI commands can be
run inside a virtual machine. This kind of software may be referred to as “SCSI target based
software”

 

Limitations of Raw Device Mapping:

+++++++++++++++++++++++++++++++++++

Not Available for Block Devices or RAID Devices
Raw device mapping (in the current implementation) uses a SCSI serial number to identify the
mapped volume. Since block devices and some direct-attach RAID devices do not export serial
numbers, they can’t be used in raw device mappings.

Not Available for Devices Attached to a Shared Adapter
If your SCSI adapter is configured as a shared adapter, you can’t use raw device mapping for any
of its devices. Only adapters dedicated to the VMkernel support raw device mapping.

Available with VMFS-2 Volumes Only
Raw device mapping requires the VMFS-2 format. If you choose not to convert your VMFS
volume from VMFS-1 format to VMFS-2 format, you cannot use raw device mapping.

No Redo Log in Physical Compatibility Mode
If you are using raw device mapping in physical compatibility mode, you can’t use a redo log
with the disk. Physical compatibility mode allows the virtual machine to manage its own
snapshot or mirroring operations. This conflicts with the SCSI virtualization objectives of physical
compatibility mode

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Virtual Compatibility Mode Versus Physical Compatibility Mode:

Virtual mode for a mapping specifies full virtualization of the mapped device. It appears to the
guest operating system exactly the same as a virtual disk file in a VMFS volume. The real
hardware characteristics are hidden. Virtual mode allows customers using raw disks to realize
the benefits of VMFS such as advanced file locking for data protection and redo logs for
streamlining development processes. Virtual mode is also more portable across storage
hardware, presenting the same behavior as a virtual disk file.
Physical mode for a raw device mapping specifies minimal SCSI virtualization of the mapped
device, allowing the greatest flexibility for SAN management software. In physical mode, the
VMkernel passes all SCSI commands to the device, with one exception: The REPORT LUNs
command is virtualized, so that the VMkernel can isolate the LUN for the owning virtual
machine. Otherwise, all physical characteristics of the underlying hardware are exposed. Physical
mode is useful to run SAN management agents or other SCSI target based software in the virtual
machine. Physical mode also allows virtual to physical clustering for cost-effective high
availability.

Raw Device Mapping with Virtual Machine Clusters
VMware recommends the use of raw device mapping with virtual machine clusters that need to
access the same raw LUN for failover scenarios. The setup is similar to a virtual machine cluster
that accesses the same virtual disk file, but a raw device mapping file replaces the virtual disk file.
The VMFS must be configured in shared access mode, to allow more than one virtual machine
to open the mapping file simultaneously.

 

Managing Raw Device Mappings

vmkfstools The vmkfstools utility can be used in the service console to do many of the same operations available in the management interface. Typical operations applicable to raw device mappings are the commands to create a mapping, to query mapping information such as the name and identification of the mapped device, and to import or export a virtual disk: The form of these commands is shown here:

vmkfstools -r ———————————>Create a Raw Device Mapping

vmkfstools -P———————————->Display VMFS File Block Details

vmkfstools -I———————————–>Clone a disk

vmkfstools -e ———————————–>Renaming a virtual machine disk

vmkfstools -q ———————————–>This option prints the name of the raw disk RDM. The option also prints other identification information, like the disk ID, for the raw disk.

vSAN memory or SSD congestion reached threshold limit

tapasmahanta1241 Comment
This is a common issue that we get in VSAN.
Examples of this alert are:

LSOM Memory Congestion State: Exceeded. Congestion Threshold: 200 Current Congestion: 201.
LSOM SSD Congestion State: Exceeded. Congestion Threshold: 200 Current Congestion: 201.

Congestion in vSAN occurs when the I/O rate of the lower layers of the storage subsystem fails to keep up with the I/O rate of the higher layers.

Local Log Structured Object Management (LSOM) is an internal component of vSAN, that works at the physical disk level (both flash devices and magnetic disks). LSOM also handles the read caching and write buffering for the components.

SSD is a cache device for a vSAN disk group.

The LSOM memory congestion state and LSOM SSD congestion state occur when vSAN artificially introduces latencies in the virtual machines in order to slow down writes to the flash device layer or layers.

Impact:
++++++++++++++
During an observed congestion period, higher virtual machine latencies occur.

Short periods of congestion might occur as vSAN uses a throttling mechanism to ensure that all layers run at the same I/O rate.

Smaller values for congestion are preferable, as higher value signifies latency. However, sustained congestion are not usual and in most cases, congestion should be close to zero.

Possible Solutions:
++++++++++++++++
If virtual machines perform a high number of write operations, write buffers could fill up on flash cache devices. These buffers must be de-staged to magnetic disks in hybrid configurations. De-staging can only be performed at a rate at which the magnetic disks in a hybrid configuration can handle.

Other reasons for congestion could be related to:

>>Faulty hardware

 

>>Corrupted or incorrectly functioning drivers or firmware

 

>>Insufficient I/O controller queue depths

 

>>Under specified  vSAN deployment

Tag Devices as SSD

tapasmahanta1241 Comment

You can use PSA SATP claim rules to tag SSD devices that are not detected automatically.

About this task :
++++++++++++++++
Only devices that are consumed by the PSA Native Multipathing (NMP) plugin can be tagged.

Procedure:

Identify the device to be tagged and its SATP.

esxcli storage nmp device list

The command results in the following information.

naa.6006016015301d00167ce6e2ddb3de11
Device Display Name: DGC Fibre Channel Disk (naa.6006016015301d00167ce6e2ddb3de11)
Storage Array Type: VMW_SATP_CX
Storage Array Type Device Config: {navireg ipfilter}
Path Selection Policy: VMW_PSP_MRU
Path Selection Policy Device Config: Current Path=vmhba4:C0:T0:L25
Working Paths: vmhba4:C0:T0:L25
Note down the SATP associated with the device.
Add a PSA claim rule to mark the device as SSD.
You can add a claim rule by specifying the device name.

esxcli storage nmp satp rule add -s SATP –device device_name –option=”enable_ssd”

You can add a claim rule by specifying the vendor name and the model name.

esxcli storage nmp satp rule add -s SATP -V vendor_name -M model_name --option="enable_ssd"

You can add a claim rule based on the transport protocol.

esxcli storage nmp satp rule add -s SATP --transport transport_protocol --option="enable_ssd"

You can add a claim rule based on the driver name.

esxcli storage nmp satp rule add -s SATP --driver driver_name --option="enable_ssd"

Reclaim the device.

esxcli storage core claiming reclaim --device device_name

Verify if devices are tagged as SSD.

esxcli storage core device list -d device_name

The command output indicates if a listed device is tagged as SSD.

Is SSD: true

What to do next:

If the SSD device that you want to tag is shared among multiple hosts, make sure that you tag the device from all the hosts that share the device.

What are the common types of congestion that are reported, and how can I address them in VSAN?

tapasmahanta124Leave a comment
The types of congestion and remedies for each type are listed below:
  1. SSD Congestion: SSD congestion is typically raised when the active working set of write IOs for a specific disk group is much larger than the size of the cache tier of the disk group. In both the hybrid and all-flash vSAN cluster, data is first written to the write cache (also known as write buffer). A process known as de-staging moves the data from the write buffer to the capacity disks. The write cache absorbs a high write rate, ensuring that the write performance does not get limited by the capacity disks. However, if a benchmark fills the write cache at a very fast rate, the de-staging process may not be able to keep pace with the arriving IO rate. In such cases, SSD congestion is raised to signal the vSAN DOM client layer to slow down IOs to a rate that the vSAN disk group can handle.

    Remedy: To avoid SSD congestion, tune the size of the VM disks that the benchmark uses. For the best results, we recommend that the size of VM disks (active working set), be no larger than 40% of the cumulative size of the write caches across all disk groups. Please keep in mind that for a hybrid vSAN cluster, the size of the write cache is 30% the size of the cache tier disk. In an all-flash cluster, the size of the write cache is the size of the cache tier disk, but no greater than 600GB.


  2. Log Congestion: Log congestion is typically raised when vSAN LSOM Logs (which store the metadata of IO operations that have not been de-staged) consumes significant space in the write cache.

    Typically, a large volume of small sized writes on a small working set can cause a large number of vSAN LSOM log entries and cause this type of congestion. Additionally, if the benchmark does not issue 4K aligned IOs, then the number of IOs on the vSAN stack get inflated accounting for 4K alignment. The higher number of IOs can lead to log congestion.

    Remedy: Check if your benchmark aligns IO requests on the 4K boundary. If not, then check if your benchmark uses a very small working set (a small working set is when the total size of accessed VM disks is less than 10% of the size of caching tier). Please see above on how to calculate the size of the caching tier). If yes, please increase the working set to 40% of the size of the caching tier. If neither of the above two conditions hold true, you will need to reduce write traffic by either reducing the number of outstanding IOs that your benchmark issues, or decreasing the number of VMs that the benchmark is creating.


  3. Component Congestion (Comp-Congestion): This congestion indicates that there is a large volume of outstanding commit operations for some components resulting from the IO requests to those components getting queued. This can lead to worse latency. Typically, a heavy volume of writes to a few VM disks causes this congestion.

    Remedy: Increase the number of VM disks that your benchmark uses. Make sure that your benchmark does not issue IOs to a few VM disks.

  4. Memory and Slab Congestion: Memory and slab congestion usually means that the vSAN LSOM layer is running out of heap memory space or slab space to maintain its internal data structures. vSAN provisions a certain amount of system memory for its internal operations. However, if a benchmark aggressively issues IOs without any throttling, it can lead to vSAN using up all of its allocated memory space.

    Remedy: Reduce the working set of your benchmark. Alternatively, increase the following settings while experimenting with benchmarks to increase the amount of memory reserved for the vSAN LSOM layer. Please note that these settings are per disk group. Also, we do not recommend using these settings on a production cluster. These settings can be changed via esxcli (see KB 1038578)  as follows:

    /LSOM/blPLOGCacheLines, default=128K, increase to 512K
    /LSOM/blPLOGLsnCacheLines, default=4K, tuned=32K
    /LSOM/blLLOGCacheLines, default=128, increase to 32K

vSAN performance diagnostics reports: “One or more disk(s) are not in active use”

tapasmahanta1241 Comment

This  issue means that one or more vSAN backend capacity disks do not see any IOs. This issue is delivered only for all-flash vSAN clusters and for benchmarks that have some read IO activity. While some capacity disks may not see read IOs for some intervals of time, the best performance is usually achieved when read IOs are spread evenly across all backend capacity disks.

Here is a list of possible remedies:

1: If you are running a 100% read workload, such as 100% sequential read or 100% random read, then it is possible that the contents of the capacity disk are cached in the write buffer of the cache tier. In this case, the reads are serviced from the cache tier and do not hit the capacity tier. Depending on the specifications of your cache tier and capacity tier storage, and the number of capacity tier disks in your disk group, it may be possible to get better read bandwidth if some read content is in the capacity tier. In this case, please increase the size of the virtual machine disks (VMDKs) to have more than a 400GB working set per disk group. In general, for an all-flash vSAN cluster, the performance of a 100% read workload is better with large working sets.

2: If your benchmark has some write IO component, you may increase the number of VMDKs so that all backend capacity disks have some object components. We recommend that any benchmark should create one VMDK for each capacity disk on the system, and concurrently drive IO to each of these VMDKs. A safe rule of thumb is to create two VMs for every disk group on your system, and create eight VMDKs for each virtual machine. Size VMDKs according to the available size of cache tier across all disk groups

3: Alternatively, it is possible that your benchmark is not issuing IOs to all the VMDKs that were created. Please check if that is the case, and correct it so that IOs are targeted to all VMDKs.

4:If you do not want to increase the number of virtual machines or the number of VMDKs, you may increase the “Number of disk stripes per object” (the default value is 1), in the vSAN storage policy with which the VMDKs were created. This number refers to the number of capacity disks across which each replica of a storage object is striped. You may choose to apply the policy to existing virtual machines, in which case all existing VMDKs will be reconfigured, and you must wait for the reconfiguration traffic to end. Alternatively, you can apply the policy manually to existing or new virtual machines/VMDKs.