When you try to start a backup for a system protected by the ShadowSnap agent, you receive the error message, "Backup failed due to I/O error on write operation from the protected system during the backup process."
In legacy versions of the Datto IRIS environment, this error message was "STC -31 on Write."
- ShadowSnap Agent
Causes of this error include:
- Networking / Samba trouble
- Server tuning variables need to be adjusted
- Bad install of the agent
- Protected volume is out of space
- System writer is not present on the protected machine
- VSS subsystem does not work
- Hardware problems on
- Bad sectors on the protected disk
- Disk errors and warnings in the event log
- ATAPI errors and warnings in the event logs
The AppAssure application can cause this error. It is incompatible with the Datto backup solution. When uninstalling this program, Datto recommends following this process to ensure that the AVolFlt family driver is properly removed. This driver has been found to remain on a protected system even if AppAssure (also known as Rapid Recovery or Replay) was previously removed.
Check Disk Health
1. Check the agent logs in the Protect tab of the Remote Web to see if backups for the server are constantly failing in the same state at the same spot on the disk.
2. To further confirm the issue, attempt a ShadowProtect backup to a NAS share on the Datto appliance. Observe whether or not the backup fails in the same state. ShadowProtect backups will fail at the same point as Datto backups if there is a problem with the disk.
3. Perform a chkdsk /r on the production machine for all volumes with the filesystem unmounted. Do not attempt this in read-only mode. If Windows has not identified corruption on the disk, a chkdsk with Windows mounted will not either. Running with the system unmounted ensures direct access to disk I/O and will provide a more accurate indication of the disk state. Even though errors might not appear in the event log, this only means the corruption has not become apparent to the filesystem yet.
4. If these steps do not solve the problem, and disk errors can be ruled out, refer to the Further Troubleshooting section below.
Check System Resources
1. Check the protected volume to make sure there is enough space to perform a backup. A minimum of 20% free space per protected volume, and 1 GB of RAM, need to be available on the production machine. If resources fall below these thresholds, backups will fail. Reducing system overhead during backups can also help prevent this issue.
0 – 70%
Use is good.
71 – 80%
Use is outside optimal limits. Monitor the volume growth and look at cleaning up unnecessary files and moving non-essential data to other storage.
81 – 100%
Volume is becoming critical. Snapshots will start to fail, disk queue lengths will increase. Create space.
Figure 1: StorageCraft disk use best-practices.
2. If the IP address of the protected agent has changed, update the protected system's address from the Datto appliance GUI. If the Datto appliance needs to follow special routing to reach the protected system, contact Datto technical support to assist with the changes.
1. Ensure that a stable route from the Datto device to the ShadowSnap agent is set up on port 25566.
2. Check for any stuck file restores or virtual machines that do not show up in the web interface. These can be destroyed as long as they are not in use.
3. Check the agent version.
4. If an update is available, upgrade the agent.
5. Check the used storage on the Datto appliance. If used storage is 85% of total capacity or higher, you may find increased reliability by lowering the amount of used storage. Also see: Backup skipped due to not enough free space.
6. Make sure that ports 25566 and 139 are open on the local Windows Firewall. 25566 will be open if the raw_agent_svc is running as part of the ShadowSnap agent. Another program may have taken the port if there was something listening on it.
7. Make sure that anti-virus exceptions exist for Program Files (x86)\StorageCraft
8. Investigate possible server tuning variables as suggested by StorageCraft
9. Check the Windows Event Logs for the error. If it does have an error, please check for and disable possible conflicting services, then uninstall and manually reinstall ShadowSnap and ShadowProtect.
10. Check network connectivity between the protected system and the Datto device. Rule out any IP address conflicts with the protected machine or the Datto device. Replace the network cable for the protected server, and try a different switch port.
11. Run the ShadowProtect Diagnostic runsupportmode.cmd tool provided by StorageCraft and consider the recommendations it makes for server tuning.
12. If backup errors persist, gather the ShadowProtect/ShadowSnap Diagnostic Logs and open a case with Datto Technical Support for further assistance.
- Diagnosing hard drives using manufacturer-specific tools (external article)