参考文档：
https://kb.vmware.com/s/article/10051?lang=zh_cn #如何定位被锁定的虚拟机
故障现象：
- 模块 Disk 打开电源失败。
无法打开磁盘“/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15/xa_15-000001.vmdk”或其所依赖的快照磁盘之一。未能锁定文件。 - 整合磁盘时出错: msg.snapshot.error-DISKLOCKED。
- Unable to consolidate virtual machine snapshots due to file lock.
- 虚拟机的“摘要”选项卡显示以下消息：需要快照整合 (snapshot consolidation required)
- cat: can't open 'win2008r2-delta.vmdk': Device or resource busy
已尝试过的操作:
A.查看快照管理器,无快照,手动整合磁盘,整合磁盘失败.
B.手动创建快照,并删除该快照,查看事件日志为快照移除失败,快照管理器无此快照,存储文件夹下多了delta.vmdk.
C.手动创建快照,然后使用删除全部,事件日志显示移除失败,存储文件夹又多了个delta.vmdk. (KB1002310)
D.手动创建快照,勾选静默文件系统,然后使用删除全部,事件日志显示移除失败,存储文件夹又多了个delta.vmdk.(KB1002310)
尝试确定被锁定的esxi主机和虚拟机
现以Unable to access a file <filename> since it is locked参考KB10051,做了如下操作排查：
1.开启esxi主机ssh，登录
ssh login to fh-dingwei hosted ESXi 172.40.10.74
2.查看当前注册到该ESXi主机的虚拟机
~# vim-cmd /vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
26 fh-dingwei [EMC-lun1] fh-dingwei/fh-dingwei.vmx centosGuest vmx-07
3.使用 WinSCP 下载[EMC-lun1] fh-dingwei/vmware.log
4.使用文本编辑器,打开vmware.log,从日志末向上查找locked关键字:
2014-01-18T04:04:18.433Z| vcpu-0| AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/4ecde6a3-5fa75820-5ace-78e7d15a4c40/fh-dingwei/fh-dingwei-flat.vmdk' locked (try 4)
2014-01-18T04:04:18.744Z| vcpu-0| AIOGNRC: Failed to open '/vmfs/volumes/4ecde6a3-5fa75820-5ace-78e7d15a4c40/fh-dingwei/fh-dingwei-flat.vmdk' : Failed to lock the file (400000003) (0x2013).
5.与日志分析出来的被锁定的磁盘文件一样。
合并快照的时候，由于无法锁定文件，因此删除快照失败：
2014-01-15T04:11:35.178Z| vcpu-0| AIOGNRC: Failed to open '/vmfs/volumes/4ecde6a3-5fa75820-5ace-78e7d15a4c40/fh-dingwei/fh-dingwei-flat.vmdk' : Failed to lock the file (400000003) (0x2013).
2014-01-15T04:11:35.178Z| vcpu-0| AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/4ecde6a3-5fa75820-5ace-78e7d15a4c40/fh-dingwei/fh-dingwei-flat.vmdk' locked (try 0)
6.接下来使用命令查看锁定该文件的MAC地址
# vmkfstools -D /vmfs/volumes/EMC-lun1/fh-dingwei/fh-dingwei-flat.vmdk
/vmfs/volumes/4ecde6a3-5fa75820-5ace-78e7d15a4c40/fh-dingwei
Lock [type 10c00001 offset 40220672 v 89, hb offset 3420160
gen 5613, mode 2, owner 00000000-00000000-0000-000000000000 mtime 1008063 nHld 1 nOvf 0]
RO Owner[0] HB Offset 3420160 5234a9bb-c84cc3ae-ab1e-78e7d15a4cdc
Addr <4, 67, 15>, gen 16, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 214748364800, nb 18272 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 4194304
#或者以下信息
[root@XAESXi01:/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15] vmkfstools -D win2008r2-delta.vmdk
Lock [type 10c00001 offset 57571328 v 3230, hb offset 3276800
gen 71, mode 1, owner 62ea7095-e416b41f-3dda-e04f439bd2b2 mtime 13515
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 48, 15>, gen 3185, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 386548109312, nb 368642 tbz 0, cow 0, newSinceEpoch 368642, zla 4304, bs 1048576
7.该MAC地址正好是fh-dingwei所在的ESXi主机的管理vmk0
注意：上面的 5234a9bb-c84cc3ae-ab1e-78e7d15a4cdc 和62ea7095-e416b41f-3dda-e04f439bd2b2，其中加黑部分为虚拟机所在的esxi主机物理网卡MAC地址。
使用esxcli network nic list命令列出当前esxi主机的物理网卡信息
# esxcli network nic list
Name PCI Device Driver Link Speed Duplex MAC Address MTU Description
------ ------------- ------ ---- ----- ------ ----------------- ---- ----------------------------------------------------------------
vmnic2 0000:00d:00.0 bnx2 Up 1000 Full 78:e7:d1:5a:4c:dc 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
# esxcli network ip interface list
vmk0
Name: vmk0
MAC Address: 78:e7:d1:5a:4c:dc
Enabled: true
Portset: vSwitch0
Portgroup: Management Network
Netstack Instance: defaultTcpipStack
VDS Name: N/A
VDS UUID: N/A
VDS Port: N/A
VDS Connection: -1
Opaque Network ID: N/A
Opaque Network Type: N/A
External ID: N/A
MTU: 1500
TSO MSS: 65535
Port ID: 33554436
#从以下指令可以确认虚拟机独占的ESXI主机信息
[root@XAESXi01:/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15] vmfsfilelockinfo -p /vmfs/volumes/VNX3200_01/xa_15/win2008r2-000001-delta.vmdk -v 10.1.8.50 -u administrator@vsphere.local
vmfsfilelockinfo Version 1.0
Looking for lock owners on "win2008r2-000001-delta.vmdk"
"win2008r2-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['e0:4f:43:9b:d2:b2']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 2 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host 10.1.8.51
MAC Address : e0:4f:43:9b:d2:b2
Host owning the lock on the vmdk is 10.1.8.51, lockMode : Exclusive
#确认锁定的
[root@XAESXi01:/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15] esxcli network ip interface list
vmk0
Name: vmk0
MAC Address: e0:4f:43:9b:d2:b2
Enabled: true
Portset: vSwitch0
Portgroup: Management Network
Netstack Instance: defaultTcpipStack
VDS Name: N/A
VDS UUID: N/A
VDS Port: N/A
VDS Connection: -1
Opaque Network ID: N/A
Opaque Network Type: N/A
External ID: N/A
MTU: 1500
TSO MSS: 65535
Port ID: 33554442
#使用vmkfstools无法克隆锁定的磁盘
[root@XAESXi01:/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15] vmkfstools -i xa_15-000001.vmdk /vmfs/volumes/VNX3200_
02/new_test_xa15/new_test_xa15_clone.vmdk -d thin
Failed to open 'xa_15-000001.vmdk': Failed to lock the file (16392).
8.手动vMotion fh-dingwei到其他主机,再次查看锁定主机:
[/vmfs/volumes/4ecde6a3-5fa75820-5ace-78e7d15a4c40/fh-dingwei] # vmkfstools -D /vmfs/volumes/EMC-lun1/fh-dingwei/fh-dingwei-flat.vmdk
Lock [type 10c00001 offset 40220672 v 90, hb offset 3420160
gen 5613, mode 2, owner 00000000-00000000-0000-000000000000 mtime 1107367 nHld 2 nOvf 0]
RO Owner[0] HB Offset 3420160 5234a9bb-c84cc3ae-ab1e-78e7d15a4cdc
RO Owner[1] HB Offset 3666432 5235d935-4de11740-868e-78e3b50f0bf8
Addr <4, 67, 15>, gen 16, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 214748364800, nb 18272 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 4194304
#或者
[root@XAESXi01:/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15] vmkfstools -D win2008r2-delta.vmdk
Lock [type 10c00001 offset 57571328 v 3230, hb offset 3276800
gen 71, mode 1, owner 62ea7095-e416b41f-3dda-e04f439bd2b2 mtime 13515
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 48, 15>, gen 3185, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 386548109312, nb 368642 tbz 0, cow 0, newSinceEpoch 368642, zla 4304, bs 1048576
9.使用 touch或者cat 命令工具查看锁定:
/vmfs/volumes/4ecde6a3-5fa75820-5ace-78e7d15a4c40/fh-dingwei # touch fh-dingwei-flat.vmdk
touch: fh-dingwei-flat.vmdk: Device or resource busy
#或者以下报错信息：
[root@XAESXi01:/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15] cat win2008r2--000001-delta.vmdk
cat: can't open 'win2008r2--000001-delta.vmdk': Device or resource busy
KB10051: If the touch * command fails with a device or resource busy message, it indicates that a process is maintaining a lock on the file or directory. This may be on any of the ESXi/ESX hosts which have access to the file. If the message is reported, proceed to the next section.
通过确定锁定的ESXI主机，通过以下指令查询锁定的虚拟机进程。
#其实本案例通过指令，发现没有虚拟机锁定。但是该文件被锁定了，无法释放，导致虚拟机无法整合快照、迁移等故障
#列出锁定的虚拟机信息其中Cartel 为虚拟机的World ID
[root@XAESXi01:/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15] lsof | egrep 'Cartel|xa_15-000001-delta.vmdk'
Cartel | World name | Type | fd | Description
#列出当前esxi主机的所有虚拟机信息
[root@XAESXi01:/vmfs/volumes/5ab0bba9-6ec9bd68-4baa-e04f439bd296/xa_15] esxcli vm process list
xa14_eHR_app
World ID: 35999
Process ID: 0
VMX Cartel ID: 35998
UUID: 56 4d 00 67 68 90 cc 9d-38 3d 2e 46 25 0c 51 4b
Display Name: xa14_eHR_app
Config File: /vmfs/volumes/5d78b386-6301c562-cf33-e04f439bd2b2/xa14_eHR_app/xa14_eHR_app.vmx
疑问和即将采取的措施如下:
10.从第8,9步,该文件应该是被78e7d15a4cdc也就是fh-dingwei原宿主172.40.10.74锁定???
11.按照KB10051部分,从清单中移除,再添加到原主机,有可能解决因为磁盘锁定无法开机的情况,但也有可能还是无法开机(该KB描述了无法开机的问题,也部分适用于磁盘锁定的情况,当前fh-dingwei开机运行中),也就是磁盘还是锁定状态,无法完成快照整合.
Locating the lock and removing it
- When you have identified the host holding the lock, unregister the virtual machine from the host.
- When successfully removed from the inventory, register the virtual machine on the host holding the lock and attempting to power it on. You may have to set DRS to manual ensuring the virtual machine powers up on the correct host.
If the virtual machine still does not power on, complete these procedures while logged into the offending host.
Note: If you cannot find the virtual machine in the host inventory in vCenter Server, open a vSphere or VI Client connection direct to the ESXi host. Check for any entry in the inventory labelled Unknown VM. If found, remove the unknown virtual machine from the inventory.
12.如果第11步清单重添加,无法解决磁盘锁定,快照整合失败的问题,则按照KB10051所说,需要重启该ESXi,释放锁定!! 可否有其他操作,如中止进程之类的?如有中止进程的替代办法,会有何风险?
Rebooting the ESX/ESXi host which is locking the files
By this stage, you have already investigated for identifiable VMkernel and Service Console processes which have maintained locks upon the required files, however an unidentified child process still maintains the lock. You have identified the server via the vmkfstools -D command in earlier steps, the lsof utility(ESX only) yields no offending processes, and no other virtual machines are locking the file.
The server should be restarted to allow the virtual machine to be powered on again.
Note: Collect diagnostic information prior to rebooting if you wish to investigate the issue further through analysis by VMware Technical Support.
Migrate the virtual machines from the server and restart it using these steps:
- Migrate or vMotion all virtual machines from the host to alternate hosts.
- When the virtual machines have been evacuated, place the host into maintenance mode and reboot it.
