Breaking Device Locks using vmkfstools

VMFS volume on the VMware ESX/ESXi host is locked due to an I/O error.

Example:

If naa.xxxxxx:1 represents one of the partitions used in a VMFS volume, you see this message when the VMFS volume is inaccessible:
volume on device naa.xxxxxx:1 locked, possibly because remote host x.x.x.x encountered an error during a volume operation and couldn’t recover.

If this issue occurs, the VMFS volume (and the virtual machines residing on the affected volume) are unavailable to the ESX/ESXi host.

In the /var/log/vmkernel.log file, you may see similar message indicating the same issue:
WARNING: LVM: 12345: The volume on the device naa.xxxxxx:1 locked, possibly because some remote host encountered an error during a volume operation and could not recover.
LVM: 11245: Failed to open device naa.xxxxxx:1 : Lock was not free

 

Use the vmkfstools command to break the device lock on a particular partition.

-B|–breaklock device

When entering the device parameter, use the following format:

/vmfs/devices/disks/disk_ID:P

You can use this command when a host fails in the middle of a datastore operation, such as expand the datastore, add an extent, or resignature. When you run this command, make sure that no other host is holding the lock.

 

  1. To break the lock:
    1. Break the existing LVM lock on the datastore by running this command:

      # vmkfstools –B vmfs device

      Note: You can also use the parameter –breaklock instead of -B with the vmkfstools command.

      From the preceding error message, this command is used:

      # vmkfstools -B /vmfs/devices/disks/naa.60060160b3c018009bd1e02f725fdd11:1

      You see output similar to:

      VMware ESX Question:
      LVM lock on device /vmfs/devices/disks/naa.60060160b3c018009bd1e02f725fdd11:1 will be forcibly broken. Please consult vmkfstools or ESX documentation to understand the consequences of this.

      Please ensure that multiple servers aren’t accessing this device.

      Continue to break lock?
      0) Yes
      1) No

      Please choose a number [0-1]:

    2. Enter 0 to break the lock.
    3. Re-read and reload VMFS datastore metadata to memory by running this command:

      # vmkfstools –V

    4. From the vSphere UI, refresh the Storage Datastores View under Configuration tab.

Note: This issue can also be resolved by restarting all the hosts in the cluster.

 

Leave a comment