Why Veeam BR Job Failed due to Change Tracking Target File Existence?

If you have any version of Veeam Backup & Replication for protecting virtual machines in your virtual environment and using vSphere 4 and later, the below problem might happen to you.

The error will be appear on backup job log file with more details. Each backup job has folder for logging at “C:\ProgramData\Application Data\Veeam\”. You might find lines same as the below line:

[27.04.2018 21:01:08] <23> Info [VimApi] Create snapshot, ref "vm-1086", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False" [27.04.2018 21:01:11] <23> Error An error occurred while saving the snapshot: Change tracking target file already exists. [27.04.2018 21:01:11] <23> Error An error occurred while taking a snapshot: Change tracking target file already exists. [27.04.2018 21:01:11] <23> Info An error has occured while freezing guest: Failed to create VM snapshot. Trying to switch to persistent snapshot.

Why Veeam BR Job Failed due to Change Tracking Target File Existence?

To find answer the question, let’s review that Veeam BR using which VMware feature and API for tracking changes and why tracking changes is important for backup applications?

What’s Change Tracking?

Veeam Backup & Replication using Changed Block Tracking (CBT) feature to take incremental backup from virtual machines. Changed Block Tracking (CBT) increases the speed and efficiency of block‑level incremental backups. Without CBT, Veeam BR or any other software has to scan VMFS to queries CBT through VADP (vStorage APIs for Data Protection) and gets the list of blocks that have changed since the last job session.

CBT will be enabled by Veeam Backup & Replication for all protected virtual machines in a backup job, it can be disabled by administrator on backup job or virtual machine as well.

For using CBT capabilities, the below items required:

The host must be ESX/ESXi 4.0 or later.

The virtual machine owning the disks to be tracked must be hardware version 7 or later.

I/O operations must go through the ESX/ESXi storage stack. So NFS is supported, as is RDM in virtual compatibility mode, but not RDM in physical compatibility mode. VMFS is supported, whether backed by SAN, iSCSI, or local disk.

CBT must be enabled for the virtual machine.

Virtual machine storage must not be (persistent or non-persistent) independent disk, meaning unaffected by snapshots.

The virtual disk must be located on a VMFS volume, backed by SAN, iSCSI, or local disk.

The virtual machine must have zero (0) snapshots when CBT is enabled, for a clean start.

Take another look at main question “Why Veeam BR Job Failed due to Change Tracking Target File Existence?”

We found that what is Change Tracking.

What’s Target File

Now, Target File!

When CBT is enabled on virtual machine, vSphere create a ctk file for each VMDK file including virtual disks and snapshot disks. The ctk files are contains the changes since last backup.

After a successful backup and full snapshot consolidation, there should be no snapshot related ctk files remaining in the datastore. Sometimes, ctk files will be remain even after successful backup and snapshot consolidation. When a bad ctk file is remains, backup application can not create new snapshot and backup will be failed.

Here is an example of this situation, I received the same error during backup of the below virtual machine. After browsing the datastore, I found a ctk file which has no flat file, so it was a orphan ctk file.

What’s Solution?

Remove orphan file: Easiest solution is removing orphan ctk file from virtual machine’s directory but the virtual machine should be powered off. Storage vMotion: Migrate virtual machine to another datastore and then remove the file from old folder and then virtual machine folder from old datastore. Disable or Reset CBT: You can disable CBT on virtual machine and remove any ctk file and enable CBT again. It’s not recommended because it will reset CBT on virtual machine and the next backup will be full backup. Read the instruction at these link: Enabling or disabling Changed Block Tracking (CBT) on virtual machines (1031873) – Also resetting CBT for virtual machine is possible Resetting Changed Block Tracking for VMware vSphere virtual machines (2139574)

Run the backup job after removing the orphan ctk file and incremental backup will be finished successfully.

Keep your virtual infrastructure software (vSphere) up to date to preventing such issues. Apply latest patches on ESXi and vCenter always.

More on Teimouri.Net

[How To]: Veeam ONE – Controlling VM Disk I/O

VM Backup Best Practices