SIRIS, ALTO, and NAS: Error: Backup failed due to I/O error from the protected system during the backup process

Follow

Issue

When you try to start a backup for a system protected by the ShadowSnap agent, you receive the error message, "Backup failed due to I/O error from the protected system during the backup process."

In legacy versions of the Datto IRIS environment, this error message was "STC -31 on Final."

Environment

  • ShadowSnap Agent

Cause

Causes of this error include:

  • Networking / Samba issues
  • Server tuning variables that require adjustment
  • Bad install of the agent
  • Protected volume is out of space
  • System writer is not present on the protected machine
  • VSS subsystem does not work
  • Hardware problems on the system
  • Bad sectors on the protected disk
  • Disk errors and warnings in the event log
  • ATAPI errors and warnings in the event logs

The AppAssure application can cause this error. It is incompatible with the Datto backup solution. When uninstalling this program, Datto recommends following this process to remove the AVolFlt family driver properly. This driver has been found to remain on a protected system even after removing AppAssure (also known as Rapid Recovery or Replay).

Resolution

Check Disk Health

1. Check the agent logs in the Protect tab of the Remote Web to see if backups for the server are consistently failing in the same state at the same spot on the disk.

2. To further confirm the issue, attempt a ShadowProtect backup to a NAS share on the Datto appliance. Observe whether or not the backup fails in the same state. ShadowProtect backups will fail at the same point as Datto backups if there is a problem with the disk.

3. Perform a chkdsk /r on the production machine for all volumes with the filesystem unmounted. Do not attempt this in read-only mode. If Windows has not identified corruption on the disk, a chkdsk with Windows mounted will not either. Running with the system unmounted ensures direct access to drive I/O and will provide a more accurate indication of the disk state. Even though errors might not appear in the event log, this only means the file system has not reported the corruption yet.

4. If these steps do not solve the problem, and disk errors can be ruled out, refer to the Further Troubleshooting section below.

Check System Resources

1. Check the protected volume to make sure there is enough space to perform a backup. The production machine requires a  minimum of 20% free space per protected volume and 1 GB of RAM. If resources fall below these thresholds, backups will fail. Reducing system overhead during backups can also help prevent this issue.

 Volume Use

Status

Comments

0 – 70%

Good

Use is good.

71 – 80%

Warning

Use is outside optimal limits. Monitor the volume growth and look at cleaning up unnecessary files and moving non-essential data to other storage.

81 – 100%

CRITICAL

Volume is becoming critical. Snapshots will start to fail; disk queue lengths will increase. Create space.

2. If the IP address of the protected machine has changed, update it from the Datto appliance GUI by following the steps described in this article. If the Datto appliance needs to follow special routing to reach the protected machine, contact Datto technical support to assist with the changes.

Further Troubleshooting

1. Ensure that a stable route from the Datto device to the ShadowSnap agent exists, and can communicate over TCP port 25566.

2. Check for any stuck file restores or virtual machines that do not show up in the web interface. You can destroy these as long as they are not in use.  

3. Check the agent version.

4. If an update to the agent is available, upgrade the agent.

5. Check the used storage on the Datto appliance. If it is 85% of total capacity or higher, you may find increased reliability by freeing up space. Also see: Backup skipped due to not enough free space.

6. Make sure that ports 25566 and 139 are open on the local Windows Firewall. Port 25566 will be open if the raw_agent_svc is running as part of the ShadowSnap agent. If another program uses any of these ports, the agent may malfunction.

7. Make sure that anti-virus exceptions exist for the StorageCraft ShadowSnap application

8. Investigate possible server tuning variables as suggested by StorageCraft.

9. Check the Windows Event Logs for the error.  If it does have an error, verify and disable possible conflicting services, then uninstall and manually reinstall ShadowSnap and ShadowProtect.

10. Check network connectivity between the protected system and the Datto device. Rule out any IP address conflicts with the protected machine or the Datto device. Replace the network cable for the protected server, and try a different switch port.

11. Run the ShadowProtect Diagnostic (external link) runsupportmode.cmd tool provided by StorageCraft and consider the recommendations it makes for server tuning.

12. If backup errors persist, gather the ShadowProtect/ShadowSnap Diagnostic Logs and open a case with Datto Technical Support for further assistance.

Additional Resources


Was this article helpful?

1 out of 1 found this helpful

You must sign in before voting on this article.

Want to talk about it? Have a feature request?

Head on over to our Community Forum or get live help.

Datto Homepage