Downloading multiple files from folder on embedded system via webservice (http, ftp, dashboard,...)

[Jürgen B.]

Hi,

I have an embedded system (Harting Mica) which is like a raspberry pi and i am writing a node-red application for it.

I would like to access/download files via the dashboard or a http/ftp request. 

My whole application is storing data and downloading/backing up the files about once a year. The files can reach 100MB and in sum all the files could reach 32GB.

The files are stored on an external drive (no sub directories, all files are directly on the drive). 

I already accessed/imported multiple files in the node-red application using the nodes fs-ops-dir and file in.

For the download part I tried: http in -> putting a file in the payload -> setting the header that it downloads the file -> http response

as described here:

https://groups.google.com/d/msg/node-red/RujsHCCtGzg/MbKUl2iXEQAJ

I could download one file, but not multiple files.

My problem is that I cannot download all files with this solution because I cannot put all files together and send it (the sum of the files would reach 32GB and my RAM is not that big). 

Therefore, I would need a solution to send file by file with one request.

Is the http request approach possible for my application and if yes, how?

Would make an ftp server in node-red make sense? 

Which other possibilities do I have to access/download the files on my external drive via webservice?

Thank you very very much in advance for your help!

Best regards,

Jürgen

[Jürgen B.]

Any advice?

Best Regards,

Jürgen

[Nick O'Leary]

Hi Jurgen,

to be honest, for that sort of bulk file transfer I'd look at well understood technologies such as rsync that can efficiently transfer multiple files over a ssh connection.

Node-RED isn't always the answer :)

Nick

On 23 November 2017 at 20:23, Jürgen B. <juergen...@gmail.com> wrote:

Any advice?

Best Regards,

Jürgen

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/0eb63ff0-a597-48c2-af26-c775c85cbd82%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jürgen B.]

The data pieces itself are not that big. They are sensor values. The reason why I mentioned 100MB per file are because I thought of storing the sensor values in one file every day, but it's probably not the best idea. In addition, I would like to do analysis with the data at some point.

This probably changes possibilities. 

Still, thanks for your answer. I'll have a look at rsync. ;)

Jürgen

[Dave C-J]

I'd definitely recommend checking rsync - it can operate over ssh secured links, can do partial updates (ie only changes) - and can recover from incomplete transfers - and can compress during transfer, - and finally can delete files once transferred if necessary. Goodness all round...