Overview
FTP is the fastest method for downloading large files (such as virtual disk exports) from the cloud.
Unfortunately, there aren’t any reliable, high performance FTP clients for Windows that are free to use.
Several commercial FTP clients are available (CuteFTP) but these are relatively expensive.
Multi-threaded vs segmented downloads
| Multi-threaded download involves multiple files being downloaded in parallel, simultaneously. Each file download runs in its own CPU process thread, independently of other downloads in progress. |
| Segmented download occurs when a single large file is broken into multiple pieces. Each piece is downloaded in parallel, simultaneously (just like a multi-threaded download.) The difference is that a segmented download applies multiple threads to a single file, which accelerates download performance. |
The FTP protocol supports specifying an offset into the file when starting a download. The offset makes it easy to resume an interrupted download - and also makes it possible to split a file into multiple parts, downloading each in parallel, starting with a different offset position within the file.
Since none of the free, native Windows FTP clients support segmented downloading, we decided to borrow an FTP client from Linux.
Say what?
What are your choices?
Windows Subsystem for Linux (WSL)
It's a little-known fact that virtually everything that runs on Linux can also be run on Windows.
Microsoft provides a Linux compatibility option called Windows Subsystem for Linux, (WSL), a relatively complex Windows add-on used to emulate Linux within a Windows environment.
WSL runs most native Linux utilities directly within your Windows environment.
However, for our purposes, Windows Subsystem for Linux is too complex.
Cygwin
Another possibility is Cygwin: an open-source project that provides a compatibility layer, allowing virtually all Linux applications to be compiled to run natively on Windows.
However, in its entirety, Cygwin is a huge project with many hundreds of gigabytes of files, containing far more content than we need to download large files from the cloud.
Introducing LFTP4WIN
LFTP is a light-weight text-based FTP client for Linux that is heavily optimized for speed and performance, including segmented downloads by default.
LFTP4WIN (an open-source project) has streamlined the delivery of LFTP for use on Windows machines by combining (a) a minimalist deployment of Cygwin with (b) an extensively customized version of WinSCP. With this combination, the project has provided an easy-to-use GUI user interface (in the form of a customized WinSCP) with the high-performance download capability of LFTP running on Windows.
TIP: Before you install LFTP4WIN
You can improve performance by increasing the number of segments
By default, LFTP uses five (5) concurrent segment connections.
If your internet is extremely fast, you may get slightly better performance by increasing the number of segments.
STEP 1. To adjust LFTP settings, click the open-lftp-conf-override option in the custom command button bar.
STEP 2. Enter a number in the Pget parts configuration and click OK to continue.
Install LFTP4WIN
LFTP Installation is performed by a Windows shell script. The script performs all actions necessary to deploy a portable, minimal Cygwin environment containing LFTP, along with a customized WinSCP client.
STEP 1. Save the install script to an empty folder on your system.
- All downloaded files are stored inside the empty folder you specify.
- No software will be installed into Windows itself.
Get the zipped installation script file here:
You will need to unzip the file after you download it.
STEP 2. Double click the script file to run it.
Double-clicking the install script will launch the installer in a cmd prompt.
STEP 3. The core files will be downloaded, and the Cygwin installer will launch.
Cygwin will copy about 250MB of files from the internet to deploy a minimal environment for running Linux utilities on Windows.
STEP 4. Once the Cygwin installer exits, the script will complete unpacking the customized scripts and utilities to run and manage LFTP from within Windows.
Once the script completes unpacking, it will then exit.
STEP 5. After the script has completed, your folder which launched the installer will now contain a portable Cygwin Linux environment, ready for running LFTP.
You may copy this folder to a USB drive or other handy location to use it anywhere.
Keep the LFTP4WIN environment updated
As with all software, you should install the latest updates to avoid security issues and to resolve known bugs.
STEP 1. To perform an update of your LFTP4WIN container, double-click the LFTP4WIN-updater.cmd script within the folder.
The Cygwin installer launch, and any new package updates will be identified and downloaded.
STEP 2. Answer the update prompts with y to continue
STEP 3. Click Next to continue.
STEP 4. The package updates will download and install.
Download files via the WinSCP user interface
STEP 1. Within the LFTP4WIN container folder, double-click this item:
DoubleClik Me – WinSCP Startup.cmd
This will open the customized WinSCP utility bundled in the package.
STEP. 2. Select New Site and configure the connection to the AxcientVault
2.1. Set File Protocol to FTP
2.2. Set Encryption to TLS/SSL Explicit encryption
2.3. Enter your FTP User name and Password
Click Login to connect.
Note: You can find your FTP username and password on the vault on the Network Sharing page.
- First, from the vault, go to the Protected System Details page
- Click Mount or Export on any snapshot. This will enable files and folders or virtual disk images to be presented for downloads.
- Next, on the Network Sharing page, check Enable FTPS Server
- Retrieve the user name and password for the FTP connection.
- Click Show Password to unhide the password when you are ready.
How to use LFTP with WinSCP
Once you have logged in via WinSCP, you are presented with the file commander view.
The left column is your local system, and the right column is the remote FTP site.
- For file and folder recovery, double click Mounted Volumes in the lower right pane and continue browsing down to the specific protected system, snapshot, volume, and file/folder you want to recover.
- For virtual disk recovery, double click on Exported VMs in the lower right pane and continue browsing down to the specific protected system, snapshot, and virtual disk image you wish to download.
Transfer files using customized LFTP scripts integrated into WinSCP
For our purposes, we are using WinSCP only for its convenient UI.
We won't use the native download and file transfer features of WinSCP.
Instead, we'll let LFTP to do the actual transfers - using customized scripts integrated within the copy of WinSCP embedded in our environment.
STEP 1. Click to select a file.
The customized command options become available in the UI:
STEP 2. To begin a download using LFTP, click
pget-to-local
in the custom commands button bar.
LFTP will open in a terminal window and display progress.
Notes:
- Speed is calculated in real-time, based on the average transfer speed for all segments.
- Units are presented in mega-bytes per second. To contrast this with your expected internet bandwidth speed, multiply by 10 to get an approximation of mega-bits per second.
- The ‘dots’ represent the full size of the file being downloaded.
- The ‘ohs’ show what portion of each segment is completed.
- All segments are downloaded simultaneously, but some segments may get better overall throughput than others, depending on the underlying data and compression.
- All segments must be completed before the file will become available.
- LFTP downloads are resumable. If the transfer is interrupted, restart the download and it will resume where it left off.
What is happening?
LFTP creates two files for each download when segmented downloads are occurring:
- One file is the partially downloaded file, with an extension of .lftp. This is a sparse file which contains blocks of data being downloaded.
- The second file contains status information about the download, including the position of each segment used when resuming an interrupted download.
LFTP joins the segments on the fly.
As soon as the download is finished, the file is ready to use and the .lftp file extension is removed.
SUPPORT | 720-204-4500 | 800-352-0248
- Contact Axcient Support at https://partner.axcient.com/login or call 800-352-0248
- Free certification courses are available in the Axcient x360Portal under Training
- To learn more about Axcient products, sign up for a free one-on-one training
- Subscribe to the Axcient Status page for updates and scheduled maintenance
1460