Overview
Starting May 17, 2021, Druva has changed the cryptographic library for VMware and Hyper-V workloads to remediate vulnerabilities as part of Druva's Patch Management Program. To equip yourself for this change, we need you to upgrade your VMware backup proxy, #########{{p hoenixagent}}s on the Hyper-V hosts, and Hyper-V FLR proxy from version 4.7.x to version 4.9.4 before May 17, 2021.
You can manually download agent version 4.9.4 for VMware backup proxy, Hyper-V hosts, and Hyper-V FLR proxy from the Downloads page.
To upgrade the agents, follow the instructions in the following articles:
Impacted environments
Workload | Configuration |
VMware | Standalone ESXi Servers and vCenter Servers |
Hyper-V | SCVMM, Hyper-V cluster, Standalone Hyper-V host Server |
All VMware backup proxy servers, Hybrid Workloads agents on the Hyper-V hosts, and Hyper-V FLR proxy servers on versions 4.7.x to 4.8.0 (excluding 4.8.0) use an older cryptographic library to securely store the credentials used while protecting VMware and Hyper-V backups. The older cryptographic library is now being replaced with a newer one to enhance security further.
Proxy server versions 4.8.0 to 4.9.4 have a combination of the old and new cryptographic libraries. Druva can use the older cryptographic library to decrypt the existing credentials and encrypt them with the newer cryptographic library.
Proxy client version 5.0.0 that goes live on 17 May 2021 will only have the newer cryptographic library and will not be able to decrypt the existing credentials.
Impact if proxies are not upgraded by 17 May 2021
If you are unable to run upgrades before 17 May 2021, you may need to perform the operations outlined below, to get backups and restores working again.
📝 Note
We recommend letting any ongoing backup or restore jobs finish before initiating the VMware proxy, Hybrid Workloads agent on Hyper-V hosts, or the Hyper-V FLR proxy upgrade.
Impact on VMware Servers
Upon an auto-upgrade from version 4.7.x to version 5.0.0, your VMware proxies will automatically get disconnected if the affected proxy was the only proxy in the backup proxy pool. The backup jobs will go into a queued state and any new restore jobs will not be initiated. If the proxy pool has more than one proxy, and one of those proxies is still connected, your backups and restores will continue seamlessly.
💡 Tip
Before you initiate an upgrade, you need to confirm if you want to proceed with the upgrade if jobs are in progress.
Impact on ongoing jobs
If the VMware backup proxy is upgraded while scheduled backups or ongoing restores are in progress, the jobs will go into a queued state. These jobs will resume once the proxy upgrade is complete.
Scheduled backups and ongoing restores for Windows VMs on VMware may fail.
Impact on Backup Now jobs (manual backups)
Backup Now jobs will fail
If you've initiated a VMware backup proxy upgrade while a Backup Now job is in progress, the job will fail with the PHOENIX189 error.
Impact on Hyper-V Servers
Upon an auto-upgrade of your Hyper-V proxy from v4.7.x to 5.0.0 , the Hyper-V backups and restores will start failing with HYPERV3 error unless the steps under Manual Steps to reconnect the backup proxy section are performed.
The following manual, scheduled, and ongoing jobs are impacted:
Backup now jobs (manual backups) may fail.
If you initiated a Hybrid Workloads agent on Hyper-V host or FLR proxy upgrade when Backup Now or FLR restores were in progress, the Backup Now or FLR restores will fail with PHOENIX189.
Scheduled backup jobs and ongoing restores for Hyper-V may fail.
Action required
You are required to upgrade the backup proxy servers from version 4.7.x to 4.9.4 before 17 May 2021.
If the proxy servers are directly upgraded from version 4.7.x to version 5.0.0, Druva will not be able to decrypt the credentials using the newer cryptographic library. This can cause the backups and restores to fail until a few manual operations are performed.
Manual steps to reconnect the backup proxy
If the backup proxies automatically upgrade from 4.7.x to version 5.0.0 and get disconnected, here are the steps to reconnect the proxy:
📝 Note
Proxy server versions 4.8.0 to 4.9.4 have a combination of the old and new cryptographic libraries. Druva can use the older cryptographic library to decrypt the existing credentials and encrypt them with the newer cryptographic library. If you directly upgrade from 4.8.0 or later to 5.0.0, you don't have to perform the following steps.
VMWare
Login to the host machine (Standalone ESXi servers or the vCenter Servers) using an account with administrative privileges.
Issue the following commands in sequence:
[root@backup-proxy-21-12-1-1 ~]# vCenterDetail set <FQDN or IP> <Username> <Password>
📝 Note
Enter the password in single quotes. For example, if your password is P@ssw0rd, in the command above, enter ‘P@ssw0rd’.
vCenter credentials set successfully
[root@backup-proxy-21-12-1-1 ~]# service Phoenix statusDruva is running
[root@backup-proxy-21-12-1-1 ~]# service Phoenix restartRestarting Druva (via systemctl): [ OK ]
[root@backup-proxy-21-12-1-1 ~]# service Phoenix statusDruva is running
[root@backup-proxy-21-12-1-1 ~]# rpm -qa | grep druvadruva-phoenix-vmware-client-5.0.0-113368.x86_64
Ensure that the Hybrid Workloads service is up and running. The backup proxy will show connected again. For more information on the commands, see Use the vCenterDetail utility.
Hyper-V
Login to the host machine (SCVMM, Hyper-V cluster, Standalone Hyper-V host Server) using an account with administrative privileges.
Issue the following command:
PhoenixHyperVControl.exe set_credential -e (scvmm/cluster) -f scvmm_fqdn (only for scvmm) -u user -p passwd
📝 Note
Execute this command from the host with elevated permissions
Restart the Hybrid Workloads Agent Client Service from the Services console.
Ensure that the Hybrid Workloads service is up and running. The backup proxy will show connected again.
Known issues
All of these are legacy issues that existed in the product before these security changes went live.
Known issue | Description |
VMware | |
The VMware backup proxy for vCenter was upgraded while backups for some VMs were in progress. The proxy upgrade and the backup jobs were both successful. Post the backup proxy upgrade, the backup of the same VMs fails with the INTERNAL65535 error.
Workaround: | |
The VMware backup proxy for standalone ESXi was upgraded. The proxy upgrade was successful. Post the backup proxy upgrade, the backup of the VMs fails with the INTERNAL65535 error however, these backups were successful before the proxy upgrade.
Workaround: | |
The VMware backup proxy for vCenter upgrade is initiated while the restores for some VMs were in progress. The VM gets into an inconsistent state, and the restore jobs fail. Post the backup proxy upgrade, the backup of the restored VMs fails.
Workaround: | |
HyperV | |
If you upgrade the Hybrid Workloads agent on Hyper-V host and the Hybrid Workloads agent restarts while a restore to the original location is in progress, the original source VM is lost. Druva deletes the source VM and then replaces it with the restored VM. The backups (full and incremental) of the source VM are lost too.
Workaround
Post the proxy upgrade, perform a restore to alternate location and take backups of the restored VM. | |
Hybrid Workloads agent upgrades on standalone Hyper-V host from version 4.8.x to 5.0.0 may intermittently fail if jobs are in progress during the proxy upgrade.
Workaround
Retry the Hybrid Workloads agent upgrade when no backup or restore jobs are in progress. | |
If Hybrid Workloads agents on Hyper-V hosts are upgraded while restore jobs are in progress, the restores may fail with the INTERNAL 65535 error.
Workaround
Initiate restores for the VMs once the proxy upgrade is successful |