Problem description
A scheduled backup job for a VMware virtual machine (VM) fails to complete within the configured backup window. The backup job starts as an incremental backup, but the agent fails to query VMware Changed Block Tracking (CBT) and falls back to a full disk scan (HCache). For VMs with multi-terabyte virtual disks, the full disk scan cannot complete within the allotted backup window, resulting in a timeout and job termination.
Cause
This issue occurs when VMware CBT fails to return changed disk areas, forcing the Druva Phoenix agent to perform a full disk scan to protect data integrity.
Common causes include:
Stale, corrupted, or orphaned CBT tracking files (
*-ctk.vmdk).The VM has reached or approached the VMware limit of 32 snapshots.
Orphaned snapshots left behind after failed backup or consolidation tasks.
Unconsolidated disks requiring manual snapshot consolidation.
Traceback
The Phoenix agent logs (cbtmanager.go) capture the failure to query changed disk areas:
level=error ts=<YYYY-MM-DD>T<HH:MM:SS>Z filename=cbtmanager.go:xxx message="Failed to query change disk area" VMDK="[<Datastore_Name>] <VM_UUID>/<VM_Name>.vmdk" Error="Error caused by file /vmfs/volumes/<Datastore_UUID>/<VM_UUID>/<VM_Name>.vmdk"
The Druva Phoenix Management Console Job Progress log reflects the switch from an incremental scan to a full disk scan prior to the window timeout:
YYYY-MM-DD HH:MM:SS | Starting backup operation YYYY-MM-DD HH:MM:SS | Actual Disk Scan type used: Incremental disk scan YYYY-MM-DD HH:MM:SS | HCache decision: Full disk scan detected - using HCache YYYY-MM-DD HH:MM:SS | Initial attempt Backup window expired YYYY-MM-DD HH:MM:SS | Finished backup operation with status: Failed
Resolution
1. Consolidate and Clean Up VMware Snapshots
Log in to the VMware vSphere Client and select the affected VM.
Check the VM summary banner. If a Configuration Issues alert states that virtual machine disks require consolidation, right-click the VM and select Snapshots > Consolidate.
Open Snapshot Manager and review existing snapshots. Ensure total snapshots do not exceed VMware limits (maximum 32; VMware recommends ≤ 2-3).
Delete any orphaned, stale, or temporary snapshots created during prior backup attempts.
2. Reset Changed Block Tracking (CBT)
Open the vSphere Web Client.
Right-click the virtual machine and click Power Off.
Right-click the virtual machine, click the Snapshot and navigate to Snapshot Manager. Ensure there are no active snapshots. If there are snapshots present, consolidate them to commit the changes. For more information, see Consolidating/Committing snapshots in ESXi.
Right-click the virtual machine and click Edit Settings.
Click the Options tab, select the Advanced section and then click Configuration Parameters.
Disable CBT for the virtual machine by setting the
ctkEnabledvalue tofalse.Disable CBT for the individual virtual disks attached to the virtual machine by setting the
scsix:x.ctkEnabledvalue for each attached virtual disk tofalse.
Where scsix:x is SCSI controller and SCSI device ID of your virtual disk.Open an ssh session to the host where the VM resides and navigate to /vmfs/volumes/datastore_name/VM_name and remove the .vmsd file along with any ctk.vmdk files in the VM folder(s). You can create a subfolder and move them there or just delete them.
In the vSphere Client, right-click the virtual machine and click Power On.
If you are using a backup application which utilizes CBT, you must re-enable CBT manually. Some backup tools automatically enable CBT. For more information, see Enabling or disabling Changed Block Tracking (CBT) on virtual machines.
Note: The first backup after resetting CBT is full. All subsequent incremental backups can reference only the change IDs generated after the full backup.
3. Adjust Backup Window for Baseline Scan Because the first backup following a CBT reset requires a full disk scan to re-index changed blocks:
In the Druva Phoenix Management Console, open the backup policy assigned to the VM.
Temporarily extend the Backup Window duration to allow sufficient time for a multi-terabyte disk scan.
Save the policy.
4. Trigger and Verify Backup
Manually trigger a backup job for the VM from the Druva Phoenix Management Console.
Confirm the baseline job completes successfully.
Restore the backup window to its original scheduled duration for subsequent incremental backups.
Verification
Review the new backup job progress log in the Druva Phoenix Management Console.
Ensure the log confirms:
Successful query of the CBT change area.
Actual Disk Scan type used: Incremental disk scan(on the second backup post-reset).Job status indicates Successful without triggering the
HCachefull scan fallback.