Overview

FTP is the fastest method for downloading large files (such as virtual disk exports) from the cloud. 

Unfortunately, there aren’t any reliable, high performance FTP clients for Windows that are free to use.  

Several commercial FTP clients are available (CuteFTP) but these are relatively expensive.

Multi-threaded vs segmented downloads

The FTP protocol supports specifying an offset into the file when starting a download. The offset makes it easy to resume an interrupted download - and also makes it possible to split a file into multiple parts, downloading each in parallel, starting with a different offset position within the file.

Since none of the free, native Windows FTP clients support segmented downloading, we decided to borrow an FTP client from Linux.

Say what?

What are your choices?

Windows Subsystem for Linux (WSL)

It's a little-known fact that virtually everything that runs on Linux can also be run on Windows.

Microsoft provides a Linux compatibility option called Windows Subsystem for Linux,  (WSL), a relatively complex Windows add-on used to emulate Linux within a Windows environment. 

WSL runs most native Linux utilities directly within your Windows environment.

However, for our purposes, Windows Subsystem for Linux is too complex.

Cygwin

Another possibility is Cygwin: an open-source project that provides a compatibility layer, allowing virtually all Linux applications to be compiled to run natively on Windows.

However, in its entirety, Cygwin is a huge project with many hundreds of gigabytes of files, containing far more content than we need to download large files from the cloud.

LFTP is a light-weight text-based FTP client for Linux that is heavily optimized for speed and performance, including segmented downloads by default. 

LFTP4WIN (an open-source project) has streamlined the delivery of LFTP for use on Windows machines by combining (a) a minimalist deployment of Cygwin with (b) an extensively customized version of WinSCP. With this combination, the project has provided an easy-to-use GUI user interface (in the form of a customized WinSCP) with the high-performance download capability of LFTP running on Windows.  

IMPORTANT: Known limitations

• Microsoft Azure will not accept virtual disk files marked as 'Sparse' for virtualization. 

LFTP4WIN may mark downloaded virtual disk image files as 'Sparse' during the download process. 

To correct this issue:

After downloading the file, open a CMD prompt using 'run as Administrator' 

Then, use fsutil to remove the sparse flag

To do this, run:

fsutil sparse setflag "<filename>.vhdx 

Replace <filename> with the path and filename of your virtual disk file

Note: In order to run successfully, the file should not be open in any program. This can be checked by using the Handle utility: https://learn.microsoft.com/en-us/sysinternals/downloads/handle. 

• To verify the change is applied correctly (and to check the status of the sparse flag),  run: 

fsutil sparse queryflag "<filename>.vhdx 

You should receive a confirmation message: 

"This file is NOT set as sparse"

TIP: Before you install LFTP4WIN

You can improve performance by increasing the number of segments

By default, LFTP uses five (5) concurrent segment connections.  LFTP Installation is performed by a Windows shell script. The script performs all actions necessary to deploy a portable, minimal Cygwin environment containing LFTP, along with a customized WinSCP client.

If your internet is extremely fast, you may get slightly better performance by increasing the number of segments.

STEP 1. To adjust LFTP settings, click the open-lftp-conf-override option in the custom command button bar.  

STEP 2. Enter a number in the Pget parts configuration and click OK to continue.

How to install LFTP4WIN

LFTP installation is performed by a Windows shell script. The script performs all actions necessary to deploy a portable,  minimal Cygwin environment containing LFTP, along with a customized WinSCP client.

STEP 1. Save the install script to an empty folder on your system.

• All downloaded files are stored inside the empty folder you specify.
• No software will be installed into Windows itself.

Get the zipped installation script file here: 

https://training.securewebportal.net/x360/x360Recover/Linux/LFTP4WIN-installer.zip

You will need to unzip the file after you download it.

STEP 2. Double click the script file to run it as normal user

STEP 3. The core files will be downloaded, and the Cygwin installer will launch

STEP 4. Once the Cygwin installer exits, the script will complete unpacking the customized scripts and utilities to run and manage LFTP from within Windows.

Once the script completes unpacking, it will then exit.

STEP 5. After the script has completed, your folder which launched the installer will now contain a portable Cygwin Linux environment, ready for running LFTP.

You may copy this folder to a USB drive or other handy location to use it anywhere.

How to use LFTP with WinSCP

Once you have logged in via WinSCP, you are presented with the file commander view.

The left column is your local system, and the right column is the remote FTP site.

• For file and folder recovery, double click Mounted Volumes in the lower right pane and continue browsing down to the specific protected system, snapshot, volume, and file/folder you want to recover.
• For virtual disk recovery, double click on Exported VMs in the lower right pane and continue browsing down to the specific protected system, snapshot, and virtual disk image you wish to download.

Transfer files using customized LFTP scripts integrated into WinSCP

For our purposes, we are using WinSCP only for its convenient UI.

We won't use the native download and file transfer features of WinSCP.

Instead, we'll let LFTP to do the actual transfers - using customized scripts integrated within the copy of WinSCP embedded in our environment.

STEP 1. Click to select a file.

The customized command options become available in the UI:

STEP 2. To begin a download using LFTP, go to the custom command bar and click

•       pget-to-local (to download a single file)
•       mirror-to-local (to download multiple files/folders)

LFTP will open in a terminal window and display progress.

Notes:

• Speed is calculated in real-time, based on the average transfer speed for all segments.
• Units are presented in mega-bytes per second. To contrast this with your expected internet bandwidth speed, multiply by 10 to get an approximation of mega-bits per second.
• The ‘dots’ represent the full size of the file being downloaded.
• The ‘ohs’ show what portion of each segment is completed.  
• All segments are downloaded simultaneously, but some segments may get better overall throughput than others, depending on the underlying data and compression.
• All segments must be completed before the file will become available.
• LFTP downloads are resumable. If the transfer is interrupted, restart the download and it will resume where it left off.

What is happening?

LFTP creates two files for each download when segmented downloads are occurring:

1. One file is the partially downloaded file, with an extension of .lftp. This is a sparse file which contains blocks of data being downloaded.  
2. The second file contains status information about the download, including the position of each segment used when resuming an interrupted download.

LFTP joins the segments on the fly. 

As soon as the download is finished, the file is ready to use and the .lftp file extension is removed.

 SUPPORT | 720-204-4500 | 800-352-0248

• Contact Axcient Support at https://partner.axcient.com/login or call 800-352-0248
• Free certification courses are available in the Axcient x360Portal under Training
• To learn more about  Axcient products, sign up for a free one-on-one training
• Subscribe to the Axcient Status page for updates and scheduled maintenance

1460  |  1815  |  1950