This article will explain how to fix a specific permissions-based issue on Windows Server 2012 R2, which causes backups to fail with a 401 or 500 error, after a random amount of days following an uninstall and reinstall of the ShadowSnap application.
As Windows 2012 servers become more popular in client environments with the older servers being upgraded and replaced a very specific problem has started arising. The behavior of this error is that the Windows server will start failing backups with a STC 401 or 500 error after doing an uninstall and reinstall of the Shadow Snap agent on the server and this behavior of reinstalling, getting backups for a few days, and then failing with a 401 error repeats itself.
Multiple Datto appliances protecting a single server
If agents are taking constant full backups or differential merges at random intervals, or are encountering intermittent 401, 500, or VSS writer errors, this is a sign that there may be more than one Datto appliance on the network trying to back up the same protected machine.
Datto does not recommend protecting a production system with more than one Datto appliance at the same time. Even if the appliances have different scheduled backup times, both systems cannot work in tandem with each other, because they both store information in the same location on the protected server with the same naming schemes, and will overwrite each others' changes to those files.
Group Policy issues
From our investigation this is caused by a group policy that is applied to the server which limits the permissions of the "Local System User" so the services started as the "Local System User" do not have full access to the Program Files directory as it does when it is first installed.
The following is what is seen by tech support on the back end and also is able to viewed through the web UI by going to the Protect tab select Backup Logs for the agent having trouble pairing, and then selecting Agent Logs.
A list of the symptoms for this specific issue follows:
- Machine is a Windows Server 2012 / R2 and commonly is the environment's Domain Controller, but this has been observed on Exchange machines as well.
- The backups start to fail with an STC error 401 or 500
- The ShadowSnap application is able to be uninstalled and reinstalled and the backups will continue to work for a limited amount of time
- The agent logs read three 500 errors followed by three 401 errors as displayed in the screenshot above.
- VSS Errors in the registry (typically) with event ID 8193 "Access is denied"
If all of these symptoms are being observed the following steps have had a very high success rate at resolving this issue.
To resolve this issue the 4 services that get installed when the ShadowSnap application gets installed have to be stopped and started under a domain administrator account. To do this please do the following steps:
- Repair Agent Communications to re-pair the agent with your Datto appliance. The ShadowProtect software will automatically attempt to reprovision its license during this process.
- Open the services windows by opening a run command and then typing: services.msc
- Stop the following services:
- ShadowProtect Service Properties
- StorageCraft Image Ready
If this service is running, change its startup type to Manual after stopping it.
- StorageCraft Raw Agent
- StorageCraft Shadow Copy Provider
- Next, right click on the each of those four services one at a time and select properties.
- In the properties window select the Log On tab.
- Next, change the Log-on As: property from Local System Account to This Account
- Please enter a DOMAIN administrator account for these services to run under by browsing the domain accounts - see the below screenshot
- After all of the services have had the domain account inserted to the log on pages, kill all related services through task manager, and start the services in the following sequence:
StorageCraft Shadow Copy Provider
StorageCraft Raw Agent