Table of Contentsx360Recover Analysis Tool (AT) Requirements to run ATWhere do I find the AT results? BootVMAutoVerify About tiered testingWhat is the difference between AutoVerify and Cloud AutoVerify?Configure AutoVerify View AutoVerify ResultsAutoVerify Global MonitoringPossible AutoVerify failure reasons RAID (Redundant Array of Inexpensive Disks)How does RAID work?RAID levelsZFS RAID vs. Typical RAIDWhy you shouldn't use RAID with x360Recover ZFS pool:
The x360Recover Analysis Tool helps determine Windows VSS readiness.
- Access to an installed x360Recover backup agent, the Recovery Center, or Axcient Direct Restore
- Outbound port 2525 must be open to the internet (Used to automatically update tickets with results.)
A x360Recover support ticket ID is optional and can be obtained by creating a ticket at https://axcient.com/partner-support/
How to run AT from the appliance or vault
1. From the web interface, click the Protected Systems tab on the left navigation.
2. Click on the System Name of the troubled protected system:
The Run Analysis Tool option will open in a popup window.
Enter the Support ticket number if you are submitting the analysis results to Axcient Support. The results from the AT will then be automatically sent to that support ticket number.Delete
4. To see the results of the analysis:
(a) Click the green download icon under the Result column or
b) If you run the AT from an appliance or vault, you can view the completed analysis results in
When the AT is run, the AnalysisTool folder is created in the current directory of the running system.
How to enable:Delete
AutoVerify is an enhancement to x360Recover's nightly BootVM check feature which provides validation of the recoverability of backup snapshots.
- AutoVerify is now part of the BootVM check operation on all x360Recover appliances and is reported within x360Recover Manager and the Global Management Portal (GMP) on v8.3.0 or greater.
- When nightly BootVM checks are enabled for each protected system on the local x360Recover appliance, each selected protected system will automatically boot the latest backup snapshot and perform AutoVerify operations to ensure that the recovered backup is healthy.
- By design, BootVM checks capture a screenshot of the protected system for manual review after it has successfully booted.
- AutoVerify extends the BootVM check functionality by automatically performing additional operations to ensure the recoverability of the snapshot. If the AutoVerify operations identify any data consistency issues within the snapshot , the operation will fail and the appliance will be notified.
Under specific conditions, AutoVerify will raise an alert on the Alerts tab and attempt to ‘self-heal’ backup consistency issues by requesting a new full backup to synchronize the protected system with the appliance. (Note: Self-healing full backups are disabled in the initial release.)
AutoVerify also provides details of protected system health status to the Axcient support team via telemetry, to enable us to proactively monitor and respond to support requests.
Benefits of AutoVerify
- The enhanced backup validation provided by AutoVerify operations allows for a high level of confidence in the integrity and recoverability of your backups. This will reduce the time spent manually fixing issues that might occur during a recovery fail over.
- Being alerted to possible issues with protected systems before they fail ensures that you have good backups and allows you to repair issues with bad data on the source system.
- You can lower the cost of ownership by automatically detecting, alerting and fixing backup consistency issues. Instead of manually reviewing BootVM screenshots on a daily, weekly or monthly basis, rely upon new automated validation and alerting.
- Reduce the time spent reviewing BootVM checks and provide faster identification of possible backup issues.
- Prior to AutoVerify, BootVM checks ran for a fixed 10-minute interval and then captured a screenshot before moving on to the next system. This could lead to either wasted time spent waiting after the VM is up, or in some cases, grabbing a screenshot too soon and returning an inconclusive check. AutoVerify proactively monitors the progress of the checks and will not exit while checks are running and the VM is responding to monitoring. (A maximum of 90 minutes is allowed for all AutoVerify tests to complete.) Once all checks are passed, AutoVerify will immediately exit and begin testing the next protected system. This provides a much more efficient boot check process and returns definitive results as to the status of the protected pystem snapshot in most cases.
- More memory may be used during Boot VM checks. Prior versions used a fixed 2GB of RAM for running checks. BootVM checks now dynamically allocate 1/4 of available RAM (with a minimum of 2GB and a maximum of 12GB) to the virtual machine to ensure there are sufficient resources to complete all tests.
About tiered testing
Axcient hosts tens of thousands of protected systems across multiple international data centers, making it challenging to perform complete AutoVerify testing on a nightly basis for every endpoint.
During the AutoVerify process, a number of tests are performed on the protected system to validate its data integrity. These tests include:
Cloud AutoVerify (for Direct-to-Cloud vault endpoints) brings all of the functionality of AutoVerify to Axcient-hosted vaults for Direct-to-Cloud endpoints. (Until v10.11, AutoVerify checks on vault endpoints could only be performed on self-hosted private cloud vaults.)
Note: is enabled only for Direct-to-Cloud endpoints on Axcient hosted vaults. AutoVerify testing for appliance-based protected systems can and should be performed on the appliance. Data replicated from an appliance to the vault is identical and there is no reason to perform redundant boot testing in the cloud for these endpoints.
will initially be monitored directly on the vault.
As of x360Recover v8.3.0, BootVM AutoVerify performs the following operations to validate each protected system snapshot status:
The appliance verifies basic communications with the running virtual machine, and if not established, fails the check. This operation confirms that the system was able to fully boot the operating system, start services, and complete plug-n-play device discovery
Check Disk (chkdsk)
A Windows ‘chkdsk’ operation (using a Windows utility that checks the integrity of a file system) is performed on each volume within the protected system.
- ‘chkdsk’ may fail due to the following conditions:
- Corrupted or incomplete backup snapshot on the BDR appliance.
(A new full backup will likely resolve the issue)
- File system corruption on the source protected system disk(s).
(‘chkdsk’ should be run on the protected system to fix the disk problem)
- Corrupted or incomplete backup snapshot on the BDR appliance.
- If chkdsk exits with specific errors codes that indicate possible data consistency issues, the appliance attempts to resolve the issue by performing a new full scan of the protected system and an alert is generated on the appliance. Alerts generated by AutoVerify are automatically closed after the full scan completes.
- Note: Full scans generated by AutoVerify will occur no more than once every 30 day
Note: ‘Self-heal’ full backups are disabled in the initial release
BootVM checks automatically allocate RAM dynamically to the virtual machine during testing. One quarter of available RAM is allocated to the VM, with a minimum of 2GB and a maximum of 12GB, in order to ensure sufficient resources to complete the AutoVerify operations.
BootVM checks end automatically. The appliance monitors the status of the AutoVerify operations and the virtual machine will be shut down as soon as all checks are complete. Note: Maximum virtual machine run time is 90 minutes, and the AutoVerify operation will fail if not completed by then.
Based on the new tiered testing structure, results are broken up into multiple parts, showing the different testing intervals.
Clicking on the AutoVerify results provides BootVM AutoVerify complete details on all operations:
In addition to appliance-level alerting, AutoVerify has been integrated into x360 Recover Manager reporting. This means you can easily see AutoVerify operations across all of your appliances and protected systems.
A summary of AutoVerify operations exists in two places:
1. On the Devices page:
2. On the Backup summary report:
AutoVerify failed to initiate
In this case, the x360Recover BDR was unable to perform a test for the protected system.Delete
The heartbeat check verifies that the Virtual Machine communication driver is able to connect to the running operating system.Delete
A common question: What is RAID, and what do all those RAID numbers mean?
RAID stands for Redundant Array of Inexpensive Disks.
At its core, RAID is a mechanism to provide some level of redundancy to a disk storage system, so that the loss of a single disk doesn’t result in the total loss of all data being stored.
For example, if the hard drive in your workstation dies, you would lose all of your data, including the running operating system, installed applications, and all of your personal files. Using RAID pools instead of single disks is a means of preventing this type of data loss. In a RAID setup, if a single disk were to fail, your data and applications would remain accessible, although at a slight performance penalty. You can then replace the failed drive and the missing data can be reconstructed from the remaining disks.
How does RAID work?
A RAID pool consists of multiple physical disks, grouped together using either software or dedicated hardware controllers, and presented to the computer system as a single logical disk.Delete
There are four types of RAID levels commonly in use: RAID0, RAID1, RAID5, and RAID6.
(RAID2-4 technically exist, but their characteristics make them unsuitable for most use cases.)
RAID0 (RAID-Zero) is the simplest form of RAID, and since it does not provide any redundancy, you could argue that it is not really RAID at all. RAID0 simply spans all of the physical disks together into a single large logical drive. The size of the resulting logical drive is equal to the sum of all drives within the storage pool. Note: Drives within a RAID0 pool can be of any size, and do not have to match the size of other disks within the pool.Delete
- RAID1 (RAID-One) is the lowest level of RAID to provide redundancy.
- Sometimes referred to as ‘mirroring’ RAID1 works by pairing two (or more, but this is uncommon) disks of equal size together and writing all data changes concurrently to both drives simultaneously. Both drives contain exactly the same data, so losing one disk implies that you have only lost one copy, and your data can safely be accessed using the remaining disk.
- With RAID1, the total size of your storage pool is equal to the size of a single disk within the pool, and all disks must be of the same size. (Technically, you can have different sized disks, but the resultant pool size will be the size of the smallest disk in the pool.)
- RAID1 writes the same data to all disks simultaneously, so there is no performance benefit for write operations. However, since the data on all disks is identical, multiple read operations can be distributed across physical storage devices, doubling the performance when reading the disk.
- RAID6 (RAID-Six) is the most complex level of RAID commonly used, and it is sometimes referred to as Advanced Data Guarding.
- RAID6 requires at least 4 or more physical disks. Similar to RAID5, you can add as many disks to a RAID6 pool as you like, but it is recommended not to exceed 12 disks.
- RAID6 stripes data across all disks similarly to RAID5, but uses a more complex algorithm to distribute parity information. With RAID6 you can sustain the loss of any two disks within the pool without losing data redundancy.
- RAID6 provides a higher level of redundancy than RAID5 at the expense of storage density and performance. Like RAID0 and RAID5, adding more disks to the pool increases performance, but since the parity calculation is heavier and there is twice as much parity data, the total performance is substantially lower.
- The total size of the logical disk created using RAID6 is equal to the total size of all disks, minus 2 drives of parity data. (e.g. (4) 2TB disks in RAID6 = 2x2TB=4TB)
Wait, what? You said there were only four types of RAID!Delete
ZFS RAID vs. Typical RAID
x360Recover is built using the ZFS filesystem to manage backup data and snapshots.Delete
ZFS is actually not just a filesystem; it's a suite of tools and a volume manager with RAID capabilities.
Instead of mixing ZFS RAID with hardware RAID, it is recommended that you place your hardware RAID controller in JBOD mode and let ZFS handle the RAID. ZFS cannot use its native capabilities to protect the data when using a hardware RAID controller, as it is not able to perform automatic self-healing unless it controls the redundancy of the disks and the data. ZFS prefers direct, exclusive access to the disks, with nothing in between that interferes.
If utilizing iSCSI, do not carve out LUNs on the SAN. Present raw LUNS to x360Recover and use x360Recoverto build your RAID5 or RAID6.
In all environments, using RAID0 in your x360Recover ZFS Pool in any fashion is not supported for production. This coincides with the pop-up message stating RAID0 is not supported in production when creating your x360Recover storage pool.