If you have added a device that contains residual data or partition information, you must remove all preexisting partition information from the device before you can claim it for vSAN use. VMware recommends adding clean devices to disk groups.

When you remove partition information from a device, vSAN deletes the primary partition that includes disk format information and logical partitions from the device.

Prerequisites:

+++++++++++

Verify that the device is not in use by ESXi as boot disk, VMFS datastore, or vSAN.

Procedure:

++++++++

1. Navigate to the vSAN cluster in the vSphere Web Client.
2. Click the Configure tab.
3. Under vSAN, click Disk Management.
4. Select a host to view the list of available devices.
5. From the Show drop-down menu at the bottom of the page, select Ineligible.
6. Select a device from the list, and click the Erase partitions on the selected disks icon
7. Click OK to confirm.

If vCenter Server is inAccessible from vSphere Web Client below are the steps to manually remove and recreate a vSAN disk group using the ESX Command Line Interface (esxcli).

To remove and recreate a disk group using esxcli commands:

Note: These steps can be data destructive if not followed carefully.

1. Log in to the ESXi host that owns the disk group as the root user using SSH.
2. Run this one of these commands to put the host in Maintenance mode. There are 3 options:

   Note: VMware recommends using the ensureObjectAccessibility option. Failure to use this ensureObjectAccessibility mode or evacuateAllData mode may result in data loss.

   • Recommended:
     • Ensure accessibility of data:
       esxcli system maintenanceMode set –enable true -m ensureObjectAccessibility
     • Evacuate data:
       esxcli system maintenanceMode set –enable true -m evacuateAllData
   • Not recommended:
     • Don’t evacuate data:
       esxcli system maintenanceMode set –enable true -m noAction
3. Record the cache and capacity disk ids in the existing group by running this command:
   esxcli vsan storage list

   Example output of a capacity tier device:
   naa.123456XXXXXXXXXXX:
   Device: naa.123456XXXXXXXXXXX
   Display Name: naa.123456XXXXXXXXXXX
   Is SSD: true
   VSAN UUID: xxxxxxxx-668b-ec68-b632-xxxxxxxxxxxx
   VSAN Disk Group UUID: xxxxxxxx-17c6-6f42-d10b-xxxxxxxxxxxx
   VSAN Disk Group Name: naa.50000XXXXX1245
   Used by this host: true
   In CMMDS: true
   On-disk format version: 5
   Deduplication: true
   Compression: true
   Checksum: 598612359878654763
   Checksum OK: true
   Is Capacity Tier: true
   Encryption: false
   DiskKeyLoaded: false

   Note: For a cache disk:

   • the VSAN UUID and VSAN Disk Group UUID fields will match
   • Output will report: Is Capacity Tier: false
4. Then remove the disk group

   esxcli vsan storage remove -u uuid

   Note: Always double check the disk group UUID with the command:
   esxcli vsan storage list

5. If you have replaced physical disks, see the Additional Information section.
6. Create the disk group, using this command:
   esxcli vsan storage add -s naa.xxxxxx -d naa.xxxxxxx -d naa.xxxxxxxxxx -d naa.xxxxxxxxxxxx

   Where naa.xxxxxx is the NAA ID of the disk device and the disk devices are identified as per these options:

   • -s indicates an SSD.
   • -d indicates a capacity disk.
7. Run the esxcli vsan storage list command to see the new disk group and verify that all disks are reporting True in CMMDS output.

Throughout the life cycle of the Virtual Machine, profile-driven storage allows the administrator to check whether its underlying storage is still compatible. The reason why this is useful is that if the VM is migrated to a different datastore for whatever reason, the administrator can ensure that it has moved to a datastore that continues to meet its requirements.

However, this not how it works in the case of VM storage policies.With vSAN, the storage quality of service no longer resides with the datastore; instead, it resides with the VM and is enforced by the VM storage policy associated with the VM and the VM disks (VMDKs). Once the policy is pushed down to the storage layer, in this case, vSAN, the underlying storage is then responsible for creating storage for the VM that meets the requirements placed in the policy.

This makes the life bit easy for the administrators.

Storage Policy-Based Management:

Deploying a VSAN datastore is different from the traditional way of mounting LUN from a storage unit or NAS /NFS devices.

In case of VSAN for better performance and availability of the VMs VSAN uses VM storage policy and a storage policy can be attached to individual VMs during deployment of the Virtual Machine.

You can select the capabilities when a VM storage policy is created.

Similarly, if an administrator wants a VM to tolerate two failures using a RAID-1 mirroring configuration, there would need to be three copies of the VMDK, meaning the amount of capacity consumed would be 300% the size of the VMDK. With a RAID-6 implementation, a double parity is implemented, which is also distributed across all the hosts. For RAID-6, there must be a minimum of six hosts in the cluster. RAID-6 also allows a VM to tolerate two failures, but only consumes capacity equivalent to 150% the size of the VMDK.

The number of Disk Stripes Per Object:

When failure tolerance method is set to capacity, each component of the RAID-5 or RAID-6 stripe may also be configured as a RAID-0 stripe.

Storage object configuration when stripe width set is to 2 and failures to tolerate is set to 1 and replication method optimizes for is not set.

Force Provisioning:

If the force provisioning parameter is set to a nonzero value, the object that has this setting in its policy will be provisioned even if the requirements specified in the VM storage policy cannot be satisfied by the vSAN datastore. The VM will be shown as noncompliant in the VM summary tab in and relevant VM storage policy views in the vSphere client. If there is not enough space in the cluster to satisfy the reservation requirements of at least one replica, however, the provisioning will fail even if force provisioning is turned on. When additional resources become available in the cluster, vSAN will bring this object to a compliant state.

Administrators who use this option to force provision virtual machines, you need to understand that although the VM may be provisioned with one replica copy,vSAN may immediately consume these resources to try to satisfy the policy settings of virtual machines.

NOTE: This parameter should be used only when absolutely needed and as an exception.

VASA Vendor Provider:

This uses the vSphere APIs for Storage Awareness (VASA) to surface up the vSAN capabilities to the vCenter Server.

VASA allow storage vendors to publish the capabilities of their storage to vCenter Server, which in turn can display these capabilities in the vSphere Web Client. VASA may also provide information about storage health status, configuration info, capacity and thin provisioning info, and so on. VASA enable VMware to have an end-to-end story regarding storage.

With vSAN, and now VVols, you define the capabilities you want to have for your VM storage in a VM storage policy. This policy information is then pushed down to the storage layer, basically informing it that these are the requirements you have for storage. VASA will then tell you whether the underlying storage (e.g., vSAN) can meet these requirements, effectively communicating compliance information on a per-storage object basis.

Enabling VM Storage Policies

In the initial release of vSAN, VM storage policies could be enabled or disabled via the UI. This option is not available in later releases. However, VM storage policies are automatically enabled on a cluster when vSAN is enabled on the cluster. Although VM storage policies are normally only available with certain vSphere editions, a vSAN license will also provide this feature.

Assigning a VM Storage Policy During VM Provisioning

The assignment of a VM storage policy is done during the VM provisioning. At the point where the vSphere administrator must select a destination datastore, the appropriate policy is selected from the drop-down menu of available VM storage policies. The datastores are then separated into compatible and incompatible datastores, allowing the vSphere administrator to make the appropriate and correct choice for VM placement.

The vSAN datastore is shown as noncompliant when policy cannot be met.

The Ruby vSphere Console (RVC) is an interactive command-line console user interface for VMware vSphere and Virtual Center. The Ruby vSphere Console is based on the popular RbVmomi Ruby interface to the vSphere API.

The Ruby vSphere Console comes bundled with the vCenter Server Appliance (VCSA) and the Windows version of vCenter Server. It is free of charge and is fully supported. We recommend deploying a vCenter Server Appliance (minimum version 5.5u1b) to act as a dedicated server for the Ruby vSphere Console and Virtual SAN Observer. This will remove any potential performance or security issues from the primary production vCenter Server. After deploying the vCenter Server Appliance, no additional configuration is required to begin using the Ruby vSphere Console to manage your vSphere infrastructure. To begin using the Ruby vSphere Console, simply ssh to the dedicated vCenter Server Appliance and log in as a privileged user.

Advantages :

++ More detailed Virtual SAN insights vSphere Web Client

++ Cluster view of VSAN while esxcli can only offer host perspective

++ Mass operations via wildcards

++ Works against ESX host directly, even if VC is down

How do you set up ??

To begin using the Ruby vSphere Console to manage your vSphere infrastructure, simply deploy the vCenter Server Appliance and configure network connectivity for the appliance. SSH to the dedicated vCenter Server Appliance and log in as a privileged user. No additional configuration is required to begin.

KB: https://kb.vmware.com/s/article/2007619

Accessing and Logging In Below you will find the steps to log in and begin using the Ruby vSphere Console (RVC):

1. SSH to the VCSA dedicated to RVC and Virtual SAN Observer usage. login as: root VMware vCenter Server Appliance root@x.x.x.x password :

2. Login to the VCSA as a privileged OS user (e.g. root or custom privileged user).

3. Login to RVC using a privileged user from vCenter. Syntax: rvc [options] [username[:password]@]hostname

Eg:

vcsa:~ # rvc root@x.x.x.x
password:

0 /
1 x.x.x.x/
> cd 1
/localhost> ls
0 Test-Datacenter
/localhost>cd 0

I will then proceed by typing ‘cd 0’ and then ‘ls’ to view the contents of my ‘Test-Datacenter’ which will then provide me with 5 more options as seen below.

The vSphere environment is broken up into 5 areas:

++ Storage: vSphere Storage Profiles

++ Computers: ESXi Hosts

++ Networks: Networks and network components

++ Datastores: Datastores and datastore components

++ VMs: Virtual Machines and virtual machine components

/localhost/Test-Datacenter>ls
0 storage
1 computer [host]/
2 network [network]/
3 datastores [datastore]/
4 vms [vm]/
/localhost/Test-Datacenter>

You can use TAB twice to view the namespace

Once you login you can use help command to check the options (add the ‘-help’ parameter to the end of the command OR you can type ‘help’ followed by what is called the command ).For example :

help vsan
help cluster
help host
help vm

Viewing Virtual SAN Datastore Capacity: “show vsanDatastore” Here is an example of using “ls” to list out datastores within the infrastructure and then using “show” to obtain high-level information on the “vsanDatastore”. Notice the capacity and free space of the vsanDatastore.

/localhost/Test-Datacenter/datastores> ls
0 datastore1: 99.21GB 0.7%
1 datastore1 (1): 99.14GB 0.1%
2 vsanDatastore: 700.43GB 17.7% 
/localhost/Test-Datacenter/datastores> show vsanDatastore/
path: /localhost/Test-Datacenter/datastores/vsanDatastore
type: vsan url: ds:///vmfs/volumes/vsan:5207cb725036c9fc-3e560cb2fb96f36d/
multipleHostAccess: true
capacity: 700.43GB
free space: 684.14GB

Virtual SAN RVC Commands Overview: https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsan/vmware-ruby-vsphere-console-command-reference-for-virtual-san.pdf

Page 14 onwards.

I am working on the rvc commands as of now, and I will update the details in my next post: Ruby vSphere Console (RVC): Part 2

A number of customers have expressed a wish to limit the amount of I/O that a single VM can generate to a VSAN datastore. The main reason for this request is to prevent a high feed VM (or to be more precise, intense IOPS application inside in a VM) from impacting other VMs running on the same datastore. With the introduction of IOPS Limits, implemented via policies, administrators can limit the number of IOPS that a VM can do.

VSAN 6.2 has a new quality of service mechanism which we are referring to as “IOPS limit for object”. Through a policy setting, a customer can set an IOPS limit on a per object basis (typically VMDK) which will guarantee that the object will not be able to exceed this amount of IOPS. This is very useful if you have a virtual machine that might be consuming more than its required share of resources. This policy setting will ensure that there are “guard rails” placed on this virtual machine so it doesn’t impact other VMs, or impact the overall performance of the VSAN datastore.

The screenshot below shows what the new “IOPS limit for object” capability looks like in the VM Storage Policy. Simply select “IOPS limit for object” for your policy, and then select an integer value for the IOPS limit. Any object (VMDK) that has this policy assigned will not be able to generate more than that number of IOPS.

Normalized to 32KB:

++++++++++++++++

The IO size for IOPS Limit is normalized to 32KB. Note that this is a hard limit on the number of IOPS so even if you have enough resources available on the system to do more, this will prevent the VM/VMDK from doing so.

Considerations:

++++++++++++

One thing to consider is that not only is read and write I/O counted towards the limit, but also any snapshot I/O that occurs against the VM/VMDK is added to the IOPS limit.

If the I/O against a particular VM/VMDK rises about the IOPS Limit threshold, i.e. it is set to 10,000 IOPS and we receive the 10,001st I/O, then that I/O is delayed/throttled.

OSFS (Object Store File System) enables VMFS volumes to be mounted as a single datastore for each host. Data on a VSAN datastore is stored in the form of data containers called objects, which is distributed across the cluster. An object can be a vmdk file, a snapshot, or the VM home folder. A reference object is also created and holds a VMFS volume and stores the virtual machine metadata files.

Logs for OSFS are captured in : /var/log/osfsd.log

How would you make a change in vSAN datastore with osfs-mkdir and osfs-rmdir?

It is not easy to create and remove any directory form VSAN  because the vSAN datastore is object based. In order to do that, special vSAN related commands exist to perform such tasks.

If you try and create a directory on VSAN you might get and error as :

# cd /vmfs/volumes/vsanDatastore

# mkdir TEST

mkdir: can’t create directory ‘TEST’: Function not implemented

How can we create/remove a directory ?