Skip to main content

Hyper-V Backup Fails with PHOENIX1 Error During VM Live Migration

Hyper-V Backup Fails with PHOENIX1 Error During VM Live Migration

Problem description

In a clustered Hyper-V environment, a scheduled or on-demand backup job for a Virtual Machine (VM) fails with the error code PHOENIX1 (Internal error. (#100000001)).

The backup job may initially proceed through the preparation phase but fails when the Druva agent attempts to detect the live-migrated VM on its target host and update the Druva Management Console with the VM's current node location.

Symptoms

  • Backup jobs fail consistently with error code PHOENIX1 in the Druva Management Console.

  • The failure occurs after the VM has live-migrated to a different Hyper-V node within the same cluster.

  • Job retries fail with the same error.

  • Stale or duplicate VM entries may appear in the Druva Management Console.

Cause

This issue occurs due to a failure in cross-node communication or stale cluster state information:

  1. WMI Cross-Node Failure: The Druva agent on the source host cannot remotely query the target host via Windows Management Instrumentation (WMI) to verify where the VM migrated.

  2. Local Credential Fallback: The error User credentials cannot be used for local connections occurs when remote WMI queries fail and the agent attempts to authenticate using local credentials, which cannot cross host boundaries.

  3. Stale Host Mappings: The cluster configuration cached by Druva does not reflect the current VM-to-host mapping due to unrefreshed state or network communication blocks.

  4. Server Rejection: When the agent sends incomplete or unverified host location data, the Druva server cannot reconcile the state and returns the generic PHOENIX1 error.

Common Triggers:

  • Windows Updates or security patches pending/partially applied across Hyper-V nodes.

  • WMI service (Winmgmt) or Hyper-V Virtual Machine Management Service (VMMS) hung or un-responsive.

  • Host firewall rules blocking cross-node WMI / RPC communications.

  • Service account credential expiration or insufficient rights across cluster nodes.

Traceback

The following log entries appear in the Druva Hyper-V agent log located at C:\ProgramData\Druva\EnterpriseWorkloads\log\hyperv\hyperv.log:

WMI connection failed with error: User credentials cannot be used for local connections VM not found for UUID = 0C43094A-E55B-4DE6-A798-8A9C655755CC Failed to update VM migration information on server. error: Internal error. (#100000001) (Error Code : PHOENIX1) Job retry attempt:1 scheduled to run after:10 minutes

Resolution

Step 1: Verify WMI Connectivity Across All Cluster Nodes

Run the following PowerShell commands from each Hyper-V host targeting every other host in the cluster to identify RPC or WMI failures:

  1. Check remote WMI service status:

    Get-Service -ComputerName <TARGET-HOSTNAME> -Name Winmgmt | Select-Object Status, StartType

  2. Execute remote WMI query for VMs:

    Get-WmiObject -ComputerName <TARGET-HOSTNAME> -Namespace "root\virtualization\v2" -Class "Msvm_ComputerSystem" -Filter 'Caption="Virtual Machine"'

  3. Inspect WMI firewall rule groups:

    Get-NetFirewallRule -DisplayGroup "Windows Management Instrumentation (WMI)" | Select-Object DisplayName, Enabled, Direction

Expected Result: Queries must succeed without access denied or connection errors.

Step 2: Ensure Core Services Are Running

On all cluster nodes, verify the following services are running with an Automatic startup type:

  • Windows Management Instrumentation (Winmgmt)

  • Hyper-V Virtual Machine Management Service (VMMS)

  • Windows Remote Management (WinRM)

Step 3: Re-apply Cluster Credentials in Druva Agent

Clear stale credentials and re-register the cluster context across all nodes:

  1. Open an elevated command prompt on a cluster node and run:

    EnterpriseWorkloadsAgent.exe hyperv configure --type cluster --user <domain\username>

  2. Ensure the designated service account belongs to both the Local Administrators and Hyper-V Administrators groups on every node in the cluster.

Step 4: Refresh VM Listings in Console

  1. Log in to the Druva Management Console.

  2. Navigate to Protect → Hybrid Workloads → Hyper-V.

  3. Select the target cluster and click Refresh.

  4. Confirm the migrated VM displays correctly under its active host.

  5. Remove any stale or duplicate VM records if present.

Step 5: Patch and Reboot Hyper-V Hosts (If WMI Persists in Failing)

If WMI cross-node queries continue to fail after verifying credentials and firewalls:

  1. Apply all pending Windows Updates on each Hyper-V host.

  2. Perform a rolling reboot of all cluster nodes (one at a time to preserve workload availability).

  3. Re-test WMI connectivity between all hosts after reboot.

Verification

  1. From the Druva Console, trigger an On-demand Backup for the affected VM.

  2. Monitor C:\ProgramData\Druva\EnterpriseWorkloads\log\hyperv\hyperv.log and verify that no PHOENIX1 or WMI connection errors occur during job execution.

  3. Perform a test Live Migration of the VM to another host node and run an on-demand backup again to confirm that cross-node migration handling succeeds.

See also

Related Articles
Did this answer your question?