At a customer today I was confronted with a situation where VMware ESX processes had log-files open that were already deleted. This can happen when logrotate was incorrectly configured, or when operational staff removed big files to clean up diskspace quickly.

However when the file is still open, you can remove the file-entry (link) to the inode, but the diskspace will not become available until all kernel references to the inode are gone (and a process having the file open counts as a reference too).

Killing the VMware processes that had the file open was not an option since we just wanted to truncate the file without impacting the guests.

There are 2 possibilities to truncate the file. The first one is very easy. First we have to find the entry in the /proc filesystem:

[root@system root]# ls -l /proc/2334/fd/ | grep delete

lrwx------ 1 root root 64 Aug 4 18:34 3 -> /path/to/vmware.log (deleted)



Now we know the file-descriptor, we can do and truncate the (unreal) symlink to the inode:

[root@system root]# echo -n >/proc/2334/fd/3

Now this procedure is safe (given that the process did not do anything with the file-descriptors in between), but it will not help with recovering the file itself (in case you wanted that).

So the longer procedure involves debugfs. By using lsof one can find the inode of the open file:

[root@system root]# lsof -p 2334 | grep delete

vmware-vm 2334 root 3u REG 8,5 43003904 46087 /path/to/vmware.log (deleted)



This gives us the original location (/path/to/vmware.log) and the inode (46087) of the deleted file. Now we have to find out on what filesystem/device this inode is from. The easiest is:

[root@system root]# df -h /path/to/

Filesystem Size Used Avail Use% Mounted on

/dev/sda5 1.7G 1.7G 0 100% /path



Beware: the below commands are not for the faint of heart. If you do something wrong you might trash your filesystem, crash your system or inadvertently kill a dozen puppies.

So we now know the device is /dev/sda5 and we can start creating a file-entry to the inode on the correct filesystem by doing:

[root@system root]# debugfs -w /dev/sda5 -R 'link <46087> /path/to/vmware.log'



Now the file entry is back, albeit it has no links referenced to it, so it is still considered to be deleted by the filesystem if the file is eventually closed.

[root@system root]# ls -l /path/to/vmware.log

-rw-r--r-- 0 vmware vmware 43003904 Apr 18 14:48 /path/to/vmware.log

Now you can access the original content, or you can truncate the file to free up diskspace.

[root@system root]# echo -n >/path/to/vmware.log



If you happen to know how one can also restore the number of links of that inode so that the file is in fact restored, please let me know. I tried using modify_inode <46087> but that did not update the inode information when leaving debugfs.