x360Recover Agent Release Information
Targeted release starting July 13 2022
x360Recover agent 2.34 is a maintenance release. The primary focus of this release is to remediate backup failures by addressing some known issues, in addition to enhancing logging of VSS failures to aid troubleshooting. This release also makes enhancements to Smart Recovery to improve system bootability in some edge cases.
Summary of enhancements
- Improved performance of open file handling
- Fixed cause of hanging frandle.exe process
- Added log entries to highlight case where agent fails authentication with server
- Added logging of successful reconnect attempts on transport layer. (Failed attempts are already logged)
- Removed backup failure in the case where a VSS snapshot is removed after transmission is completed but before server-side backup integrity checks are completed
- Added support for booting MBR / Gen1 systems where boot partition is FAT/FAT32
- Replaced ‘Get Queued Completion Failed’ log messages with more detailed description of what failed and what needs to be completed
- Returned more thorough messaging from efsvss when VSS snapshot failures occur
- Removed requirements for hardware AES-NI support with autodetection and failback to software encryption
- Handled lack of shadow storage space where space is available and restricted only by settings
Summary of bug fixes
- Fixed issue with excluded volumes not being updated until after service restarts
- Fixed case where corrupted / missed files in backup might occur with FastDelta during a crash-consistent (Level 3) backup of the initial base snapshot (rare)
- Fixed case where backup agent was crashing due to unexpected state in Bitlocker data
- Fixed failure to boot using Smart Recovery in several edge cases found during escalations
- Fixed incorrect reporting of VSS utilities found in empty folders by the Analysis Tool
- Fixed Analysis Tool ‘Problems-log’ file always being empty
Fixed displayed volume size on the appliance being doubled for mirrored dynamic disk volumes**
**ADDENDUM: Known issues with dynamic disks
Due to a previously unknown bug with dynamic disks, the underlying disk volume size captured by previous agents has been off by a small amount. The affected disk area is outside of data storage, which means no data has been lost.
However, the corrected reporting of the mirrored dynamic disk sizes (noted above) now requires a full backup for all dynamic disks. This is needed to correct the base image and to capture the missing header data. No data duplication will occur as a result of this full backup, but a full backup will appear in the logs for systems with dynamic disks. This may also result in a slightly longer backup cycle, affecting the next backup immediately following the agent update on the protected system.
Note: x360Recover full backups taken after the initial base image has been completed are actually just "full scan" backups. No data will be transmitted to the server that is already present. Only change data not yet sent to the server will be transmitted. All data will be read from the protected system volumes and deduplicated locally before being transmitted.
Volume Shadow Copy Services (VSS) storage
We've found that two of the most frequent causes of failed backups are (a) failure to create a Volume Shadow Copy Services (VSS) snapshot or (b) deletion of that VSS snapshot by Windows during the backup. A significant number of backups are failed simply because VSS is configured with a restrictive limit on storage space (even when there is plenty of free space remaining on the volume.)
What's causing this VSS issue?
VSS is the core feature that allows capturing an image, or ‘snapshot’, of a system frozen in time during a backup.
VSS operates by first quiescing disk write operations, then returning a bitmap of the ‘frozen’ filesystem for the backup engine to process. While ‘frozen’, new data is written into free space allocated by the VSS engine, outside the in-use areas reported to the backup engine. The longer that the VSS snapshots remain active, the more ‘free’ space is silently consumed on the filesystem. If VSS runs out of available free space (either because it has not been allowed to expand beyond a set limit, or because the drive is genuinely out of space), Windows will automatically revoke snapshots, from oldest to newest, to free space available for file operations.
A substantial number of backups are failing simply because VSS settings have been configured to restrict available shadow storage space. This restrictive setting is wholly unnecessary, as Windows automatically manages shadow storage space and always gives precedence to live file system writing over shadow storage.
To resolve this issue, Agent 2.34 automatically reconfigures shadow storage limits when a backup fails due to lack of shadow storage space. Important Note: Due to reported issues with this option from some partners, Agent 2.35+ will disable this feature by default.
AES-NI support (for local cache)
For local cache to be enabled, previous versions of the agent required protected systems to have hardware AES-NI CPU instruction support. This was due to the backend encryption layer for the data.
We've found that Hypervisor platforms default to outdated CPU models (in order to enhance functional support for clustering dissimilar hardware hosts.) This is a nuisance if you are configuring local cache on virtual guests.
Enhancements to our core encryption libraries have now enabled automatic detection of hardware AES-NI instruction support (along with failback to software encryption.)
Note: For best performance, we still recommend configuring virtual machines to use a CPU platform supporting AES-NI in hardware - but the lack of AES-NI will no longer be a blocker for deploying local cache.
The Analysis Tool is our next generation replacement for the Replibit Analysis Tool.
- Completely rewritten in C++
- Collects log files and configuration information about installed Axcient x360Recover components, system event log information, system configuration details and installed software
- Optionally uploads the collected details directly to an open support ticket to assist our support team in troubleshooting your backup related problems.
For requirements and details, please review Analysis Tool (AT)
What's new with Smart Recovery?
Smart Recovery is our next generation recovery mechanism, which makes virtualizing and recovering your protected systems as bulletproof as possible.
During a recovery, Smart Recovery collects instance-specific details from your protected system physical disks, partitions, and volumes, and then returns exactly-matched virtual disk images.
Disk signatures, volume UUID’s, and even dynamic disk configurations are returned exactly like the original protected system disks.
Smart Recovery is the recommended mechanism when you are performing a recovery and is available when
- you are virtualizing a protected system
- during AutoVerify boot VM checks
- when you are exporting disks via iSCSI
Note: Smart Recovery is not yet available when performing disk exports from a vault or appliance. If you need a virtual disk export using Smart Recovery, perform a virtual disk export using Recovery Center (with or without a local cache.) Recovery Center can attach directly to an appliance for local recovery beginning with appliance version 10.18.0