Nifi template using ListFTP and FetchFTP

[Mathieu Marie]

Hi,

I modified my 'Ingest Data' Nifi template to also support FTP.
As I want to be able to pickup only the new files on the server, I am using ListFTP and FetchFTP instead of GetFTP processor.

However when I use both processors, I end up with twice the same information to enter (see attached screenshot)

How can I make sure that a user only needs to enter once the FTP connection information?

I understand that I can reference values from:
* the applicaiton.properties file like `${config.hdfs.ingest.root}`, but I can not add FTP servers in the Kylo config anytime someone has a new datasource.
* the metadata service like `${metadata.category.systemName}`, but it requires to have  the `Metadata Service` as a parameter.

Is there also a way to reference the values from ListFTP in the FetchFTP configuration?

Thanks for your help,
Regards,
Mathieu

[Greg Hart]

Hi Mathieu,

You would have to write a custom UI plugin to accomplish what you want. Generally I would recommend using the GenerateFlowFile and UpdateAttribute processors but this would only work if ListFTP allowed an incoming FlowFile which it does not.

https://github.com/Teradata/kylo/tree/master/samples/plugins/example-ui-get-file-processor-template

[Mathieu Marie]

Hi Greg,

Thanks for the pointer.
I've not be able to find many information related to the custom UI plugins? For example, the README on github does not explain how to package it, how to deploy it, etc.

Regards,
Mathieu

[Greg Hart]

Hi Mathieu,

The sample plugin uses Maven and can be packaged with 'mvn package'. You can then copy the jar file into /opt/kylo/kylo-ui/plugin/.

[Mathieu Marie]

Hi Greg,

Many thanks for the information.

Regards,
Mathieu