The backup job stalled longer than X seconds and was cancelled

Follow

Problem

Backups fail with the error message:

The backup job stalled longer than 900 seconds and was cancelled

No recent error reported from ShadowSnap agent

Cause

This error message might display a different timeout value if you have set a custom threshold in Configure Agent Settings → Advanced → Snapshot Timeout.

The ShadowSnap Agent Service was not able to start the backup process because the OS's Volume Shadow Copy Services could not take a snapshot Shadow Copy of a volume(s) in under 15 minutes.

This happens when another VSS-Aware process, such as a Scheduled Shadow Copy, runs at the same time as the ShadowSnap Agent Service, causing the VSS Writers to stall or fail.

This could also be caused by a full OS drive.

Solution

1. Check to see if the OS drive is full. If so, free up some space to allow the VSS writers to run.
2. Check the current state of the VSS writers:

  • Launch a Windows Command Prompt session as an Administrator.
  • Find each of the VSS writers in a failed state by running the command vssadmin list writers. The command-line tool will display the status of the writers, as shown in Figure 1.


Figure 1: Example output of vssadmin list writers.

  • If the vssadmin list writers command hangs and does not return any output, this indicates that the Volume Shadow Copy service itself, or one of its dependent tasks, might be in a bad state, causing the VSS writer audit to fail. Restarting this service will typically clear this state. If the service will not restart, a reboot of the production machine is likely required. A reboot is often only a temporary solution.

3. Make note of any failed VSS writers.
4. Clear the VSS writers, following the relevant method for your version of Windows.

VSS writers are a Microsoft product. If VSS writer failure continues after following the steps in this article, you will need to contact Microsoft for further troubleshooting assistance.

Repairing VSS Writers in Windows Server 2003 / XP

1. Open services.msc and stop any VSS-related services that are currently running, including:

  • Background Intelligent Transfer Service
  • MS Software Shadow Copy Provider
  • Microsoft Software Shadow Copy Provider
  • Volume Shadow Copy
  • SQL Server VSS Writer
  • Windows Backup
  • Hyper-V Volume Shadow Copy Requester

2. Download Volume Shadow Copy Service SDK 7.2. Save the vshadow.exe executable to %windir%\system32\. This will allow you to run the vshadow command from all directories of the system via the command prompt.

 
Figure 2: Volume Shadow Copy Service SDK 7.2 installer.

3. Open a Command Prompt session as an Administrator and reset the VSS writers by running the vshadow command, as shown in Figure 3. Note that the volume letter used should match that of the system's volume letter.



Figure 3: vshadow.exe

4. Run the vssadmin list writers command again to confirm that the VSS writers are now in a stable and ready state.

5. Retry a backup.

Repairing VSS Writers in Windows Server 2008, Windows Vista and Newer

1. On the production machine, open an elevated command prompt with administrator privileges.

2. Run the command vssadmin list writers.


Figure 4: Example output of vssadmin list writers.

3. Make note of any failed VSS writers.

4. Open services.msc on the production machine. Stop the services for the VSS writers that have failed.

5. Open Task Manager and end any remaining processes for the failed VSS writers.

6. Restart the services through services.msc.

7. Run vssadmin list writers again to make sure the VSS writers are now showing as stable.

Best Practices

It is imperative that all VSS writers be in a stable state on the protected machine. If there are any writers that are failed, they can cause backups to fail.

  • Make sure that there are no other backups running at the same time as your scheduled Datto backups. This includes SQL backups.
  • For long term stability and reliability for backup performance, remove all conflicting VSS-aware processes. The following table lists some of the more commonly seen services, as well as whether they conflict with Datto backups:
RedXMark.jpg  Acronis Nonstop Backup Service
RedXMark.jpg  Acronis Scheduler2 Service
RedXMark.jpg  Acronis Sync Agent Service
GreenCheck.jpg  Background Intelligent Transfer Service
RedXMark.jpg  Backup Exec Error Recording Service
RedXMark.jpg  Backup Exec PureDisk Filesystem Service
RedXMark.jpg  Backup Exec Remote Agent for Windows
RedXMark.jpg  Backup Exec VSS Provider
GreenCheck.jpg  Block Level Backup Engine Service
GreenCheck.jpg  Hyper-V Volume Shadow Copy Requestor
GreenCheck.jpg  Microsoft Software Shadow Copy Provider
RedXMark.jpg  MozyHome Backup Service
RedXMark.jpg  MozyPro Backup Service
RedXMark.jpg  Replay 5 SnapToVM
RedXMark.jpg  Replay Agent
RedXMark.jpg  Replay Core
RedXMark.jpg

 Replay XML Command

RedXMark.jpg

 SAAZCore (Current RMM Version no longer requires this, can be removed)

GreenCheck.jpg  ShadowProtect Service
GreenCheck.jpg  StorageCraft Raw Agent
GreenCheck.jpg  StorageCraft Shadow Copy Provider
RedXMark.jpg  SQL Server (BKUPEXEC)
GreenCheck.jpg  SQL Server VSS Writer
GreenCheck.jpg  Volume Shadow Copy
RedXMark.jpg

 Windows Backup

RedXMark.jpg

EVault Software BUAgent

 

If these steps do not solve the problem, changing the Backup Engine Option on the Datto appliance's Configure Agent Settings page to STC Backup Engine will typically allow a backup to go through even with some VSS writer issues on the machine.

Making the change to the STC Backup Engine will result in a crash-consistent state backups of your machine, as the STC Backup Engine is not application-aware. This will also prevent log truncation for Exchange and SQL servers, as they require the VSS writers to truncate the logs.


Figure 5: Backup engine options

Additional Resources

VSS Explained: Common Writers, Services, And Hotfixes Available




Was this article helpful?

3 out of 4 found this helpful

You must sign in before voting on this article.

Want to talk about it? Head on over to our Community Forum!