Download File From Sftp Server Using Spring Boot
Danielle Dinunzio
The Basic Spring Integration Example shows how to read files from an SFTP Server, if the data is configured with an application-context.xml file. Where is the example where a Spring Boot configuration is used, and then the code to read from that server, and the code for the test?
This just stems from my lack of knowledge on how to convert from using a traditional application-context XML file WITH annotations in the code, to using a complete Spring Boot application context configuration file.
In this blog post, we are going to see how to setup a SFTP Server using Java and Spring boot. We will be using a library/framework called as Apache MINA which provide us with the possibility to create different kinds of network applications, in our case as we want to create a SFTP server, we will be using the library Apache SSHD . This library provides us a way to support SSH protocols in our application.
Also we will be using Spring Boot with Maven to demonstrate the project.
In the line 28, we are creating a host private key in a file called as host.ser . If you are creating a fully functional SFTP interface you should usually store this file in a secure location, where it can only be accessed by the administrator.
NOTE: Once you have created the server and started accepting connections from the client, make sure that you do not loose/destory the host.ser file, this file is used to authenticate by the client to make sure that it is connecting to the right host or not.
In line 29, we create a SftpSubsystemFactory, to add SFTP capabilities to our SSH server.
In line 30, we create a simple username and password based authentication mechanism, in our case all the incoming connections should provide username and password while starting the SFTP session. Here we are using a hardcoded username and password for demonstration, in the next secion we will see how to enable Public Key Authentication for our SFTP server.
Line 31, starts our SFTP server followed by a simple log statement to tell us that the server is started.
As this example is based on Spring Boot, this is how our main class looks like:
Note that we added a statement while(true) to keep running the application, in this way it continues to run and listen to the incoming connections. So we just created a very minimal but functional SFTP server, lets test this.
To start an sftp session, I am going to use git bash, as I am using a Windows machine. Here is the command I am going to use:
SFTP can be used to transfer files to and from SFTP servers using the SFTP Protocol. To connect, specify the RemoteHost;. service uses the User and Password and public key authentication (SSHClientCert).Choose an SSHAuthMode and specify connection values based on your selection.
Set the following connection properties to control the relational view of the file system:
- RemotePath: Set this to the current working directory.
- TableDepth: Set this to control the depth of subfolders to report as views.
- FileRetrievalDepth: Set this to retrieve files recursively and list them in the Root table.
JSch supports multiple channels (operations) over a connection to the server.By default, the Spring Integration session factory uses a separate physical connection for each channel.Since Spring Integration 3.0, you can configure the session factory (using a boolean constructor arg - default false) to use a single connection to the server and create multiple JSch channels on that single connection.
The inbound channel adapter first retrieves the file to a local directory and then emits each file according to the poller configuration.Starting with version 5.0, you can limit the number of files fetched from the SFTP server when new file retrievals are needed.This can be beneficial when the target files are large or when running in a clustered system with a persistent file list filter, discussed later in this section.Use max-fetch-size for this purpose.A negative value (the default) means no limit and all matching files are retrieved.See Inbound Channel Adapters: Controlling Remote File Fetching for more information.Since version 5.0, you can also provide a custom DirectoryScanner implementation to the inbound-channel-adapter by setting the scanner attribute.
Sometimes, file filtering based on the simple pattern specified via filename-pattern attribute might not suffice.If this is the case, you can use the filename-regex attribute to specify a regular expression (for example, filename-regex=".*\.test$").If you need complete control, you can use the filter attribute to provide a reference to a custom implementation of the org.springframework.integration.file.filters.FileListFilter, which is a strategy interface for filtering a list of files.This filter determines which remote files are retrieved.You can also combine a pattern-based filter with other filters (such as an AcceptOnceFileListFilter, to avoid synchronizing files that have previously been fetched) by using a CompositeFileListFilter.
Starting with version 5.0, the inbound channel adapter can build sub-directories locally, according to the generated local file name.That can be a remote sub-path as well.To be able to read a local directory recursively for modification according to the hierarchy support, you can now supply an internal FileReadingMessageSource with a new RecursiveDirectoryScanner based on the Files.walk() algorithm.See AbstractInboundFileSynchronizingMessageSource.setScanner() for more information.Also, you can now switch the AbstractInboundFileSynchronizingMessageSource to the WatchService-based DirectoryScanner by using setUseWatchService() option.It is also configured for all the WatchEventType instances to react for any modifications in local directory.The reprocessing sample shown earlier is based on the built-in functionality of the FileReadingMessageSource.WatchServiceDirectoryScanner, which uses ResettableFileListFilter.remove() when the file is deleted (StandardWatchEventKinds.ENTRY_DELETE) from the local directory.See WatchServiceDirectoryScanner for more information.
You should consider two properties when configuring inbound channel adapters.max-messages-per-poll, as with all pollers, can be used to limit the number of messages emitted on each poll (if more than the configured value are ready).max-fetch-size (since version 5.0) can limit the number of files retrieved from the remote server at a time.
When using the recursive option (-R), the fileName includes any subdirectory elements and represents the relative path to the file (relative to the remote directory).If you use the -dirs option, each recursive directory is also returned as an element in the list.In this case, we recommend that you not use the -1 option, because you would not be able to distinguish files from directories, which you can do when you use FileInfo objects.
Migration of data or files moving around different servers is a common case in software development.
For java and spring boot based solutions, jsch implementations of SFTP are often used. In the following blogpost we will learn about the knowhows in usage and setup of most commonly used java libraries for file uploads and file downloads from remote servers.
SFTP is a secure protocol for file transfers and its fully complaint with the SSH authentication functionalities. This protocol makes authentication on both the server and the user and offers cryptographic hash functions. Appliances of this protocol are file transfers which guarantee data integrity such as migrations of documents or logs, copying data across various servers inside or outside an organization, making automated system backups etc. Everywhere where the file security is integral, this protocol should be used over FTP.
Apart from supporting uploading file to a remote server, Jsch also offers service method for downloading file from a remote SFTP server. Downloading file is supported by the get() method, which is also very easy to use.
FileSystemManager interface is used to create file objects, which are then used as arguments in the copyFrom() method. FileSystemManager is used to locate a FileObject by name from one of those file systems. FileObject is a file, and is used to access the content and structure of the file. The method copyFrom() copies another file, and all its descendants, to the method caller. On another note, the sftpUrl is a construct which consist of the s @theRemoteHost:22/andThePathToTheFile
In this blogpost, we covered two things. We covered how to setup mock SFTP server with docker and also, covered how to use SFTP integration java libraries to securely transfer files from/to remote servers. Write me in the comments sections if you find this useful or if you have further questions.
In most cases, we don't have any ability to change the FTP server. If we want to be notified of any changes in a remote file system, our client needs to connect, scan the directory, and compare it with an earlier, known state. Basically, the client computes the delta and publishes an event. But wouldn't it be nice if the FTP server could broadcast an event when something happens? That way, there can be no doubt about what happened. And there's no doubt that we observed every change. If we were using any other FTP server, this would be more of a wish than a possibility. But as we're using the Apache FTP Server, Spring Integration offers us some interesting possibilities. We can install an FTPlet, kind of like a filter, that will broadcast any important events on the FTP server as ApplicationContext events. Then, we can use Spring Integration to publish interesting events as messages that we can process in Spring Integration. This capability is a new feature in Spring Integration.
df19127ead