Skip to main content
All CollectionsKnowledge BaseEnterprise WorkloadsTroubleshooting - Enterprise Workloads
Oracle Phoenix Backup Store backup job is in the queued state or stuck in running state
Oracle Phoenix Backup Store backup job is in the queued state or stuck in running state
Updated over 7 months ago

Problem description

Oracle Phoenix Backup Store backup job is stuck at 0% or in a queued state on the Phoenix Management Console.

Cause

This error is observed when you delete snapshots manually from Phoenix Backup Store before backups of the snapshots is uploaded to Druva Cloud. The backup is stuck while searching for the deleted snapshot on Phoenix Backup Store.

Phoenix Management Console displays the number of the snapshots pending for upload more than the actual number of snapshots residing on the Phoenix Backup Store.

Traceback

Phoenix.log

+++Snip+++

level=warn ts=2021-03-29T07:32:19.987170718Z filename=utils.go:93 message="Error occured while querying for snapshot" SnapDetails=Phoenix/PBS_UPG@UID_713 message="exit status 1"

level=info ts=2021-03-29T07:32:19.987194169Z filename=backup_agent.go:478 message="Snapshot not created yet. Waiting for 5 seconds"

level=warn ts=2021-03-29T07:32:24.990219634Z filename=utils.go:93 message="Error occured while querying for snapshot" SnapDetails=Phoenix/PBS_UPG@UID_713 message="exit status 1"

level=info ts=2021-03-29T07:32:24.990261295Z filename=backup_agent.go:478 message="Snapshot not created yet. Waiting for 5 seconds"

level=warn ts=2021-03-29T07:32:29.99231142Z filename=utils.go:93 message="Error occured while querying for snapshot" SnapDetails=Phoenix/PBS_UPG@UID_713 message="exit status 1"

level=info ts=2021-03-29T07:32:29.992360423Z filename=backup_agent.go:478 message="Snapshot not created yet. Waiting for 5 seconds"

level=warn ts=2021-03-29T07:32:34.994384456Z filename=utils.go:93 message="Error occured while querying for snapshot" SnapDetails=Phoenix/PBS_UPG@UID_713 message="exit status 1"

+++Snip+++

Resolution

  1. Check the UID on the current running job on the Poenix Management Console or detailed logs.
    Progress logs from the Phoenix Management Console

    +++Snip+++
    Mar 30, 2021 22:29:11RMAN backup is starting for the Mount:TEST_UPG, Snapshot:UID_715
    +++Snip+++

    Phoenix.log
    +++Snip+++
    level=info ts=2021-03-29T07:32:19.985236108Z filename=backup_agent.go:473 message="Waiting for Snapshot to be created"
    level=warn ts=2021-03-29T07:32:19.987170718Z filename=utils.go:93 message="Error occured while querying for snapshot" SnapDetails=Phoenix/PBS_ TEST @UID_715 message="exit status 1"
    +++Snip+++

  2. Log in to Phoenix Backup Store and run the command zfs list -t snapshot to show the list of commands.

    +++Snip+++
    root@ip-10-175-209-83:~# zfs list -t snapshot
    NAME USED AVAIL REFER MOUNTPOINT
    Phoenix/PBS_ TEST @UID_717 140M - 9.23T -
    Phoenix/PBS_ TEST @UID_718 659K - 9.23T -
    Phoenix/PBS_ TEST @UID_719 32K - 9.23T –
    +++Snip+++

  3. Compare the UIDs of the snapshot from phoenix.log and on the Phoenix Backup Store from the above output. From the above example you can see that UIDs for both the snapshots are not the same and there is a difference of two snapshots, which means that snapshot UID_715 and snapshot UID_716 is deleted from Phoenix Backup Store before backing up to the Druva Cloud.

  4. Verify the pending snapshots on Phoenix Backup Store on the Phoenix Management Console.

    In this example, the pending snapshots on the Phoenix Management Console shows five snapshots because two snapshots with UIDs as UID_715 and UID_716, are deleted manually and three snapshots are available on Phoenix Backup Store with UIDs as UID_717, UID_718, and UID_719, because Druva Phoenix updates the successfully-backed up snapshots on the Phoenix Management Console.

    T1.png
  5. Verify if PBS_ TEST is mounted or not during backup which should be mounted during backup using from df -h.

    In the following example, you can see that PBS_DB1 is mounted. It is mounted because the backup is running for this DB but not PBS_ TEST. Backup is triggered but stuck because Druva did not find the snapshot UID_715 on Phoenix Backup Store.

    root@ip-10-175-209-83:~# df -h
    Filesystem Size Used Avail Use% Mounted on
    udev 16G 0 16G 0% /dev
    tmpfs 3.1G 916K 3.1G 1% /run
    /dev/nvme0n1p1 30G 12G 18G 39% /
    /dev/loop5 100M 100M 0 100% /snap/core/10908
    tmpfs 3.1G 0 3.1G 0% /run/user/1000
    Phoenix 4.9T 128K 4.9T 1% /Phoenix
    Phoenix/PBS_DB1Mount 12T 6.7T 4.9T 58% /Phoenix/ PBS_DB1

  6. Cancel the stuck job and restart the Phoenix Backup Store services. Phoenix pickups snapshot with UID_716. Follow this process till it matches snapshot UIDs that are available on Phoenix Backup Store. In this example, you need to follow twice as two snapshots are deleted manually before the backup.

    • Run the following command to restart the Phoenix Backup Store service:
      sudo service PhoenixBackupStore restart


📝 Note
The maximum snapshots that will be available on Phoenix Backup Store would be five.


Verification

The next subsequent job should be successful.

Did this answer your question?