Take care – Express Patch 6 for ESXi 6 can break your Backup (CBT Bug)!

Update 05/08/2016: The issue is fixed with ESXi 6.0 Patch 03: VMware ESXi 6.0, Patch ESXi600-201608401-BG: Updates esx-base, vsanhealth, vsan VIBs (2145664)

—- —– —– ——

Update 25/06/2016: VMware published a KB article covering the issue described in this blog post two days ago (VMware KB 2145895). If you read the KB and the post carefully, you will find some differences. Eg. VMware states in the KB: “No data is lost. Due to this issue, incremental backups are effectively full backups.” As I have seen various affected virtual machines where the backup was not equal to a full backup, I decided not to change the blog post. I still recommend performing a CBT reset to ensure that you can rely on your backup.

—- —– —– ——

Just a short notice if you are already on ESXi 6:

There is an issue with Change Block Tracking (CBT) in Express Patch 6 for ESXi 6 (Build 3825889) that can affect your backup if your backup software relies on CBT.

affected are only VMs with Operating System Server 2008 or newer (Linux VMs are unaffected.)

all vHW Versions (7, 8, 9, 10 and 11) and all VMware tools versions are affected

the issue occurs with thick and thin provisioned disks

the issue only occurs when the backup software uses quiesced snapshots

Error messages:

You can find errors in the vmware.log of the affected VMs similar to:

vcpu-0| I120: SNAPSHOT:SnapshotBranchDisk: Failed to acquire current epoch for disk /vmfs/volumes/

vmdk : Change tracking is not active for this disk 572.

Other symptoms:

you have to backup an abnormal amount of data (it is almost a full backup even if it is an incremental one)

longer backup durations due to the abnormal amount of data sent to the backup software

Resolution:

There are currently 2 workarounds available:

If you already have affected virtual machines, migrate them to a host using an unaffected ESXi version (eg. VMware ESXi 6 Update 2, build 3620759).

Then you have to reset CBT of the affected VMs and perform a Full Backup to ensure that your backup is reliable (How to reset CBT on multiple VMs using PowerCLI)

Here you can find a short description: How to rollback/revert to a previous ESXi Version

Additional information: