PROBLEM
As part of a storage replacement programme, we had migrated all our VMs off the NFS datastore. When attempting to unmount the datastore from the host(s) we where given this error:
Call “HostDatastoreSystem.RemoveDatastore” for object “datastoreSystem-122544” on vCenter Server “vcenterFQDN” failed.
An unknown error has occurred.
Checking the /var/log/vmkernel.log on the host we saw the following (where <datastorename> is the name of the NFS datstore we were trying to remove):
2017-09-29T10:38:05.498Z cpu22:59182 opID=cf1be666)WARNING: NFS: 1921: <datastorename> has open files, cannot be unmounted
Of course first thing was to check all the VMs were actually moved, yes they were. Next checking HA to make sure its not using a handle for its datastore heartbeating; again no this had all been pointed at the new storage, but i temporarily disabled HA on the cluster with no avail. Perhaps it was some of the VMs had snapshots open when they were migrated, again no this doesn’t leave file handles open on migration.
SOLUTION
Finally came across these settings on the host where it was still referencing the old storage, so they needed to be updated to point at a folder on the new storage and hey presto I could unmount the storage as expected.
- Advanced Settings -> ScratchConfig.ConfiguredScratchLocation
(Reboot of host required after change)
- Advanced Settings -> Syslog.global.logDir
All you need to do is create new log folders on the new storage for these hosts and make the settings changes on each host, after a reboot (for the scratchconfig in particular) you’ll be able to dismount the datastore.
Note that I could not find any open handles from any of the hosts to the storage using any of the tricks (shown below), it just seemed that having that value in the storage was enough to stop it dismounting as expected.
