What is LFTP
LFTP is a command-line program for several file transfer protocols. LFTP is designed for Unix and linux type of operating systems and it is distributed under GNU General Public License, so we are free to install and use.
LFTP can transfer files via FTP, FTPS, HTTP, HTTPS, FISH, SFTP, Bit Torrent and FTP over HTTP proxy. It also supports the File eXchange Protocol (FXP), which allows the client to transfer files from one remote FTP server to another.
The features of LFTP's are transfer queues, segmented file transfer, resuming partial downloads, mirror directories, bandwidth throttling, and recursive copying of file directories. The client can be used interactively or automated with scripts.
As part of my job, we deal with pulling large number of files from remote systems as and when the data is available, using java based sftp/ftp pull using JSCH library. This is also good for most of the scenarios but for every other different data collection we had to write bunch of code on top of the common JSCH based framework to collect data, mainly because of the directly structures and number of files we need collect are way different from one and other.
One of the cool future which caught my eye and perfect solution for the above mentioned problem is LFTP mirror.
