Agent event logs and alerts - x360Recover

Written By Tami Sutcliffe (Super Administrator)

Updated at November 15th, 2022

How do I use these alerts to resolve issues?

Have you noticed a message like this on the Jobs page of your vault?

These alerts appear whenever one of your protected systems is experiencing a troubled backup job. 

The purpose of these alerts is to help you troubleshoot the issue. You have the ability to both identify and resolve many of these issues which appear under normal circumstances.

  • To determine the best way to resolve a particular issue, you'll need to identify the error type on the Jobs page of your vault and then cross-reference to that error in the table below. 
  • Once you have found your specific error in the table below, follow the included Action needed for resolving that type of error.

Identify the error type and cross reference that error in the table below:

Log in to your x360Recover dashboard and navigate to the Jobs page of your vault.

As you review the Jobs page in your vault, you'll notice a variety of status alerts:

Roll over the desired status icon to read the identifying details of that specific error.

Example #1:

Rolling over this particular icon on this client, 

the following pop up is displayed:

You'll then find this error in the table below, to determine the best next steps to follow:

Example #2:
Rolling over this particular icon on this client, 
the following popup is displayed:
You'll then find this error in the table below, to determine the best next steps to follow:

Note: If you are unable to identify or cross-reference your specific error, please submit a Support ticket and we will assist you in resolving the issue.


CROSS REFERENCE TABLES:

Agent logs

Log messages and agent status as of agent 2.36:

Error Events

Event ID

Message

Action needed

501

Summary: Failed Backup - Could not create snapshot


Message "Failed Backup - Could not create snapshot\n <include efsvss details for the failure here>"

 

Perform troubleshooting related to general Microsoft Volume Shadow copy Services failures.

 

See: VSS Troubleshooting Guide
or General VSS Troubleshooting

502

Summary: Failed Backup - Error reading data from disk


Message ""Failed Backup - Error reading data from disk, one or more snapshots have been deleted during backup
 (Error = <x> Message = <x>), one or more snapshots have been deleted during backup (Error = <x> Message = <x>)"

 

Windows or some other process has deleted the snapshot being used to perform the backup.

 

Ensure that there is sufficient free space on the drive available to VSS Storage.     It may be necessary to adjust VSS Storage limits or configure storage to another volume with more available free space.

 

See the VSS Settings section in Best Practices


503

Summary: Failed Backup - Error reading data from disk during backup


Message "Failed Backup - Error reading data from disk during backup
 (Error = <x> Message = <x>) (Error = <x> Message = <x>)"

 

The agent encountered errors reading data from physical disk during the backup.

 

Perform chkdsk repair with surface scan to check for bad disk blocks and repair file system corruption on the system


504

Summary: Failed Backup - Could not resume backup


Message "Failed Backup - Could not resume previously running backup. (snapshotDetails.txt not found)"

 

A previously paused backup could not be resumed because the agent could not find the index of previous VSS snapshots in use.  Was it deleted?

 

Trigger a new backup manually or wait for the next scheduled backup interval


505

Summary: Failed Backup - Could not resume backup


Message "Failed Backup - Could not resume backup (One or more snapshots from the previous backup were not found)"

 

A previously paused backup could not be resumed because the original VSS snapshot could not be found.  Likely it was removed by  Windows due to Shadow Storage space size limits.

 

If the issue persists through multiple backups, you may need to reconfigure VSS storage limits.

 

See See the VSS Settings section in Best Practices


506

Summary: Failed Backup - Could not resume backup


Message "Failed Backup - Could not resume backup (could not open one or more snapshots required to continue backup)"

 

A previously paused backup could not be resumed because the original VSS snapshot could not be opened.  A dependent snapshot may be missing, or a snapshot may be corrupted.

 

If the issue persists through multiple backups, you may need to clear all VSS snapshots to purge the corrupted snapshot data.

 

See Backup Jobs Stuck or Hung


507

Summary: Failed Backup - Could not send data, server storage pool is full


Message "Failed Backup - Could not send data, server reports that the storage pool is full."

 

The storage pool on the Backup server is full.  Expand the storage pool or free space by adjusting retention or deleting Protected System.

 

See Snapshot Retention Settings


508

 Summary: Failed Backup - Backup Integrity Check on Server has Failed


Message "Failed Backup - Backup Integrity Check on Server has Failed. Data will be sent again on the next backup."

 

 

 

Agent alerts

Typical alert entries generated by  x360Recover agent 2.36

Alert Title

Description

Action needed

Successful Backup

A backup has completed successfully

Nothing to do.  Typically, this alert type is not enabled.  It can be useful in conjunction with ConnectWise for auditing and reporting of backup status over time

Missed Backup

The agent failed to start a scheduled backup on time

 Possibly the agent is offline, or still running a previously started backup.  Missed backup status will be automatically cleared on the next successful backup


The agent failed to start a scheduled backup on time


Failed Backup

The agent failed to successfully complete a backup

The backup may have been  interrupted, failed a backup complete check, or  had other issues.  Failed backup status will be automatically cleared on the next successful backup.

Check the agent logs, Windows Event viewer, or Backup Jobs page on the appliance for more details.

 

Partition layout change detected: automatically scheduling a full volume scan to detect any new data (no action required)


The agent has detected that the disk partition layout has changed.  A new full backup will be triggered automatically to sync disk layout changes with the appliance

Nothing to do. The next backup performed with be a full scan backup which will autocorrect the issue.

Failed to load some volume information

This alert is raised if there is a specifically configured disk volume specified for backup that the agent cannot access or cannot find on the system

Investigate the protected system and identify why the missing volume is not found.  Possibly a disk has gone offline, removable media has been removed, or a disk volume drive letter assignment has changed


Volume <Volume ID> could not be mounted

One or more volumes selected for backup by the agent could not be mounted during the backup

Investigate the protected system and identify why the specified volume(s) can not be mounted or accessed by the agent.  Possibly the device is failed, or a Bitlocker disk is not unlocked

 

Multiple volumes found with the same identity (<ID>) for <PS Name> on <Appliance Name>

Volume tracking has identified that two volumes configured for backup are the same volume

 

                

Investigate the protected system and identify the duplicate volume.  Possibly an iSCSI mount from an appliance or a volume from the local system is mounted via Recovery Center

 

 

Informational events

Event ID

Message

100

Summary:  Backup is started - Mode: [Full | Incremental | No LSN Incremental]" (i.e. Starting Backup - Mode: Full)"


Message "Backup started at <date/time> \n\nIncluded Volumes = <Backup_volumes>\nTarget backup server = <server address>\n
 Cloudserver = <Cloudserver address>\n Backup Mode = [Full | Incremental | No LSN Incremental]"

 

101

Summary:  Backup has been paused by the appliance


Message "The backup has been paused. Backup will be resumed when the schedule allows or when manually resumed from the Web UI."

 

102

Summary:  Backup paused due to agent shutdown


Message "The running backup has been paused because Agent services are shutting down. Backup will be resumed when agent services are next started."

 

103

Summary: Backup job is being resumed.


Message "The previously running backup job is being resumed."

 

104

Summary: Backup job completed successfully.



 


Backup troubleshooting tips

The following steps will help you diagnose issues you may experience with x360Recover.

Note: If you follow these instructions and are still unable to resolve the issue yourself, we are standing by to lend a hand. 

Please see the section Escalate recovery troubleshooting later in this article for details on how to open a Support ticket and run our Analysis Tool.

1. Basic troubleshooting with aristos.log 

On the protected system, open the aristos.log. 

  • Scroll to the bottom, to locate the newest entries. 
  • Note that this log continually updates. 
  • To renew the log information, close and reopen it.

How to recognize a successful backup

The beginning of a backup begins with the following line: 

INFO: Loading configuration from: C:\Program Files (x86)\Replibit\aristos.cfg 


A successful backup ends with the following line:

INFO: Backup job completed successfully 

and you will see INFO: Main function returned with: 0

You will then see a succession of INFO: Main function returned with: -1. 

The -1's are counting off until the next backup. 


How to identify errors

If you do not see the backup successfully end, search (ctrl f) for Error

Note that 99% of errors are Windows-based. (This is often because the agent calls to the Windows Volume Shadow Copy Service (VSS) for a shadow copy to process.)

Common errors are:

ERROR CAUSE
Snapshot creation for volume failed

VSS will not or cannot run
Get queued completion failed 

VSS cannot complete because some process is stuck or failing
VSS_E_PROVIDER_VETO 

VSS was (a) cut off by another program stealing VSS to run its own backup or (b) the AV/firewall killed the shadow copy.
VSS_E_INSUFFICIENT_STORAGE 


The shadow copy storage needs to be increased to 15% of the volume in order to be backed up
VSS_E_NESTED_VOLUME_LIMIT


The agent will not back up user profile disks (UPDs) or mapped drives

Where to look for contributing events/errors

Once an error is found, check the following:

  • date/time in the Windows Event Viewer
  • Windows application log
  • system log

You can see from these sources which Windows events/errors may be contributing to VSS failure to complete. These events could be related to system IO, AV/firewall, permissions or hard disk issues. 


If you are experiencing registration issues:

  • Check license availability, to ensure you have sufficient licenses to register the agent. Refer to Appliance-based license billing
  • If network connectivity is an issue, determine if the agent is able to communicate with the appliance. If networking errors or other issues interfere with this communication process, the agent will not successfully register with the appliance. Refer to Install an agent

2. Antivirus exclusions

Notes:

  • Some anti-virus systems will delete the agent files.
  • Should an exclusion require a file path, you can create a text document for axecomsvc.exe and axecom.exe in C:\Windows\System32, then rename each .txt file as .exe

Ensure that C:\Program Files (x86)\Replibit contains .exe files  and then add the exclusions below.


Exclusions to be created for the backup agent in the antivirus firewall:


Files to exclude in the Replibit directory:

  • The entire folder C:\Program Files (x86)\Replibit
  • The following files within C:\Program Files (x86)\Replibit
    • ManagementService.exe
    • UpdaterService.exe
    • agentservice.exe
    • efsvss*.exe (there are 6 files with a different version for each type of system)
    • uefi-check.exe
    • FastDelta.exe
    • xcloud-analysis-tool.exe
  • Allowlist the domain efscloud.net

Files to exclude that are not in the Replibit directory. (These files are injected post-backup during the boot or live virtual machine.:)

  • %systemroot%\system32\axecomsvc.exe
  • %systemroot%\system32\axecom.exe

How to escalate troubleshooting


After you have performed your own investigation (by following the suggested processes above), you may still have issues to be resolved.

If you are unable to resolve the issues yourself, you'll need to (a) open a new Support ticket and (b)  run our Analysis Tool

Instructions on how do perform these two important steps are below:


2.2. Open a new Support ticket from the dashboard of your x360Portal account. 

2.2.1. As you create the new Support ticket, remember to record the unique ticket number. (You will need to input that specific Support ticket number when you run the Analysis Tool later. Instructions on entering the ticket number in the Analysis Tool are below.) 

2.2.2. Be sure to mention in the new Support ticket that you plan to run the Analysis Tool. 



2.3. Run the Analysis Tool from x360Recover in your x360Portal

From the appliance or vault user interface:

2.3.1.  Click the Protected Systems tab on the left navigation.

2.3.2.  Click on the System Name of the troubled protected system.

2.3.3.  In the Protected Systems Details page, click Run Analysis Tool.

2.3.4.   When requested, input the Support ticket number you recorded earlier

 2.3.5.  Finally, click Run to complete the process. 

 

 


 SUPPORT      | 720-204-4500 | 800-352-0248

1289  |  1319