Problem description
Disaster Recovery (DR) restore jobs intermittently fail for virtual machines (VMs) that are mapped to a Turbotier (CloudCache). The DR restore job completes successfully only when all Recovery Point (RP) blocks have been completely synced to the Druva Cloud. If one or more data blocks reside exclusively on the local Turbotier (CloudCache) and have not yet finished syncing to the cloud, the DR restore job fails.
Cause
The DR restore engine requires all RP blocks to be available directly in the Druva Cloud environment to assemble and restore the virtual machine. When a restore job runs before Turbotier (CloudCache) has flushed all pending data blocks to the cloud, the DR process cannot access those local blocks, resulting in an internal S3 upload/state validation failure.
Traceback
[2025-04-21 23:06:00,441] [INFO] Builtin<ImdsV2Required> Fetching aws region using imdsv2 [2025-04-21 23:06:00,445] [INFO] Boto3<get_region_name>: successful: us-east-1 [2025-04-21 23:06:00,448] [INFO] CloudDRAgent: Upload to s3 state: {'/Mon Apr 21 16:25:38 2025/VirtualMachines/data_6000C293-d0cb-c9fb-2304-cbd239dd1d8c': {'status': 'failed', 'uploaded_size': 7724859392, 'block_size': 0, 'error_msg': 'Failed to Upload', 'rtype': 'INCR'}, '/Mon Apr 21 16:25:38 2025/VirtualMachines/data_6000C29e-8120-3af1-342b-506b1edbd631': {'status': 'failed', 'uploaded_size': 304715137024, 'block_size': 0, 'error_msg': 'Failed to Upload', 'rtype': 'INCR'}} [2025-04-21 23:06:00,448] [ERROR] CloudDRAgent: Phoenix Internal Error : Upload to s3 state is marked as failed for RP = /Mon Apr 21 16:25:38 2025/VirtualMachines/data_6000C293-d0cb-c9fb-2304-cbd239dd1d8c [2025-04-21 23:06:00,449] [ERROR] Error <class 'Exception'>: Phoenix Internal Error : Upload to s3 state is marked as failed for RP = /Mon Apr 21 16:25:38 2025/VirtualMachines/data_6000C293-d0cb-c9fb-2304-cbd239dd1d8c. Traceback -Traceback (most recent call last): File "clouddragent_py3/drRestore/drRestore.py", line 1431, in _process_uploaded_s3_keys Exception: Phoenix Internal Error : Upload to s3 state is marked as failed for RP = /Mon Apr 21 16:25:38 2025/VirtualMachines/data_6000C293-d0cb-c9fb-2304-cbd239dd1d8cResolution
To ensure DR restore jobs complete successfully, ensure that all Recovery Point (RP) blocks are fully flushed from Turbotier (CloudCache) to the Druva Cloud prior to triggering or scheduling a DR restore job.
Recommended Action & Workaround:
Maintain a minimum 4-hour gap between the Turbotier (CloudCache) flush schedule and the DR restore job schedule. This buffer allows sufficient time for all RP blocks to upload to the cloud.
Update Turbotier (CloudCache) Flush Schedule:
Log in to the Druva Console.
Go to Manage and select Turbotier (CloudCache).
On the Overview page, navigate to the Configuration tab in the left panel.
Click Edit to modify the flush schedule.
Adjust the timing to guarantee at least a 4-hour buffer prior to any scheduled DR restore operation.
Update DR Restore Job Schedule:
Adjust the DR plan schedule so that restore execution takes place after the 4-hour flush window has elapsed.
Verification
Verify via the Console interface that the Turbotier (CloudCache) flush/sync status displays as completed for the associated backup set.
Trigger or wait for the next scheduled DR restore job.
Check the job execution logs to confirm that the DR restore completes with a Success status without logging
Upload to s3 state is marked as failed.