File Upload to NR?

[Julian Knight]

Hi all,

Does anyone have an example of uploading a file from a web page into Node-Red?

I can do a post easily enough using a form like:

<form action="/upload", method="post", enctype="multipart/form-data">

  <input type="file", name="displayImage">

  <input type='submit'>

</form>

But when received by an http-in node, the data is embedded in the body and it needs to be extracted from the form data.

Seems as though this should be simple but after searching through this group and elsewhere, I can't find any examples other than raw node.js ones.

[Julian Knight]

Nobody uploading files to NR then?

[Nicholas O'Leary]

HI Julian,

we don't include the necessarily middleware that will automatically parse the body response and extract the files for you. Something for the todo list when we move to Express 4.0 in node-red 0.12 (more about that when I'm back from holiday next week).

Despite being on holiday, had a quick poke at what's possible today.  It is just about possible, but is a bit convoluted and isn't ideal...

1. in the directory that contains your settings.js file ($HOME/.node-red probably), run:

        npm install connect-multiparty

2. edit settings.js, add the following line at the top (outside of the module.exports = {} block)

        var mp = require('connect-multiparty')();

3. depending how recent your settings.js file is, there should be a httpNodeMiddleware setting commented out. If not, add one like this:

       httpNodeMiddleware: function(req,res,next) {

           mp(req,res,next);

       },

This inserts the appropriate file upload parsing middleware into the stack of middleware used by the http nodes. The file information is then stored in msg.req.files - throw that at the debug node to see what it looks like.

The middleware I've used for this, pretty much the first I found, https://github.com/andrewrk/connect-multiparty comes with a couple warnings as it creates temporary files for the uploads and doesn't delete them - leaving that to you. There may be a better choice of middleware - but the principle is there.

Nick

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Knight]

Hi Nick,

Thanks for responding, didn't mean for you to interrupt your hols, take a break man!

Thanks for pointing me in the right direction, should help sort something out. I had a short window to sort something out so I picked up a previous project of mine and have tweaked it. But I really want to revisit this at some point.

If anyone is interested, I also worked out a rough and ready way to pass JSON data across to an spawned PowerShell command. It wasn't straightforward.

[Dave C-J]

I used the multer npm to achieve the same.... So yet another option :-)

[Stephan Wissel]

So you did?

var multer = require("multer");

httpNodeMiddleware: function(req,res,next) {

           multer(req,res,next);

       },

[Kuan Yu Chen]

Hi all,

As for the node-red 0.12 that update in November, does there exist any other better ways to handle the data received from the client just like Julian wanted to do?
Majorly, I want to upload a image at a web page(client side), and finally transfer it into a base64 string(server side).
I had tried to display the payload form the http-in, and I got a buffer. And then, I tried the .toString() method over that buffer, and I got some information along with gibberish about the image.
Because I just tried to use the node-red recently, If there are any other better ways to upload a file from a web page, please tell me. Thank you very much.

Nick O'Leary於 2015年8月6日星期四 UTC+8上午3時55分19秒寫道：

[Greg EVA]

You could consider doing this all outside of Node-RED in the interim.

I once setup a file uploader using some HTML/JS which I found which uploaded a file to a public incoming folder on the web server (appropriate public R/W permissions obviously).

You could then use the exec node to run standard Linux commands in a small bash, PERL, Python, etc. to check for new files in the folder and convert them to base64.

I'm not sure about this part, but you may be able to recover the flow using a file node which is looking for the newly created base64 files.

[Jonathan Nakandala]

How were you able to use multer as the middleware for http in nodes?

I've been trying to find instructions past what's on this page but my unfamiliarity with node/express adds to the difficulty.

[Mattias Marder]

Hi,
Did anyone here accomplish this task without accessing the file system?

The options suggested above all require writing a temporary file to disk.

An option that doesn't write to disk is a package called skipper but I couldn't get it to work as a httpNodeMiddleware. However outside node red, the skipper does it job well.

[Nicholas O'Leary]

Hi Mattias,

We certainly want to add http file upload to our core node set, and doing it without writing to a temporary file would be nice... although in some circumstances, a file would be better as you wouldn't want to be passing the file contents around the flow too much.

From Orit's previous posts to the group asking about skipper, it has been on my to-do list to investigate properly - but my focus has been on getting 0.14 finished. We'll be releasing 0.14 either tonight or tomorrow. If you want to send me what you've tried with skipper, I can take a look at it this week.

Nick

--
http://nodered.org

Join us on Slack to continue the conversation: http://nodered.org/slack

---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.

To post to this group, send an email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

[Dave C-J]

The other to consider/compare is multer.

[Orit Prince]

Hi Nick

Thanks for helping !
Here are the samples:

1. NR with skipper

-npm install skipper
-npm install skipper-memory
-npm install streamifier
-settings.js:
... httpNodeMiddleware: function(req,res,next) {
//next();
skipper(req,res, next);
},
functionGlobalContext: {
skipperMemory: require('skipper-memory'),
streamifier: require('streamifier')
},
-flow:
[{"id":"38f3acb4.787ad4","type":"function","z":"c3e758bf.055ed8","name":"multipart form-data parsing","func":"node.log(\"hi there\");\n\nvar skipperMemory = global.get('skipperMemory');\nvar streamifier = global.get('streamifier');\n\nvar tempFolder = \"/temp\";\n\n//Upload attachement to mem storage\nvar memFileName = new Date().getTime().toString();\nnode.log(\"memFileName: \" + memFileName);\n\nif (msg.req.file){\n node.log(\"found msg.req.file\");\n}else{\n node.log(\"not found msg.req.file\");\n}\nmsg.req.file('file').upload({\n adapter: skipperMemory,\n dirname: tempFolder,\n filename: memFileName\n}, function (err, uploadedFiles) {\n if (err) {\n msg.payload = err;\n\t\tnode.error(\"Error\", msg);\n }\n else {\n node.log(\"uploaded to mem cache: \" + uploadedFiles[0].fd);\n \n //Retrieve attachement from mem cache \n skipperMemory.read(uploadedFiles[0].fd, function(err, content){\n if (err){\n msg.payload = err;\n node.error(\"Error\", msg);\n }\n else{\n node.log(\"retrieved from mem cache: \" + uploadedFiles[0].fd);\n \n //Add attachement to formData\n msg.headers = msg.req.headers;\n formData.attachments = streamifier.createReadStream(content);\n \n //Delete attachement from mem cache\n skipperMemory.rm(uploadedFiles[0].fd, function(err, content){\n if (err){\n msg.payload = err;\n node.error(\"Error\", msg);\n }\n else{\n node.log(uploadedFiles[0].fd + \" removed from mem cache\");\n node.send(msg);\n }\n }); \n }\n });\n }\n});\n","outputs":1,"noerr":0,"x":649,"y":150,"wires":[["42033b30.95a444"]]}]

- Pasted here please find the 'multipart form-data parsing' function script with no white spaces:

node.log("hi there");
var skipperMemory = global.get('skipperMemory');
var streamifier = global.get('streamifier');

var tempFolder = "/temp";

//Upload attachement to mem storage
var memFileName = new Date().getTime().toString();
node.log("memFileName: " + memFileName);

if (msg.req.file){
node.log("found msg.req.file");
}else{
node.log("not found msg.req.file");
}
msg.req.file('file').upload({
adapter: skipperMemory,
dirname: tempFolder,
filename: memFileName
}, function (err, uploadedFiles) {
if (err) {
msg.payload = err;
node.error("Error", msg);
}
else {
node.log("uploaded to mem cache: " + uploadedFiles[0].fd);

//Retrieve attachement from mem cache
skipperMemory.read(uploadedFiles[0].fd, function(err, content){
if (err){
msg.payload = err;
node.error("Error", msg);
}
else{
node.log("retrieved from mem cache: " + uploadedFiles[0].fd);

//Add attachement to formData
msg.headers = msg.req.headers;
formData.attachments = streamifier.createReadStream(content);

//Delete attachement from mem cache
skipperMemory.rm(uploadedFiles[0].fd, function(err, content){
if (err){
msg.payload = err;
node.error("Error", msg);
}
else{
node.log(uploadedFiles[0].fd + " removed from mem cache");
node.send(msg);
}
});
}
});
}
});


2. Node.js app with skipper as Middleware and skipper-memory adapter

skipper_server.js:
==================
var express = require("express"),
app = express();
var async = require('async');
var fs = require('fs');
var skipperMemory = require("skipper-memory")();
app.use(require("skipper")());
var appEnv;
startServer = function(){
app.listen(appEnv.port, function() {
console.log('listening on port', appEnv.port);
});
}

var configFile = process.argv[2] || 'mem_settings.json';
fs.readFile(configFile, 'utf8', function (err, data) {

if (err) {
logger.error(err);
process.exit(1);
}

appEnv = JSON.parse(data);
startServer();
});

var tempFolder = "/temp";

app.get("/files", function (request, response) {
skipperMemory.ls(tempFolder, function(err, fileNames){
if (err){
response.status(500).send(err);
}else{
console.log(JSON.stringify(fileNames));
response.json(fileNames);
}
});
});

app.get("/latest", function (request, response) {
skipperMemory.ls(tempFolder, function(err, fileNames){
if (err){
response.status(500).send(err);
}else{
console.log(JSON.stringify(fileNames));
if (fileNames.length > 0){
skipperMemory.read(tempFolder + "/" + fileNames[fileNames.length - 1], function(err, content){
if (err)
response.status(500).send(err);
else{
response.setHeader('content-type', 'image/png');
response.send(content);
}
});
}else{
response.json({status: "empty cache"});
}
}
});
});

app.get("/file", function (request, response) {
console.log("get file " + JSON.stringify(request.query));
skipperMemory.read(request.query.fd, function(err, content){
if (err)
response.status(500).send(err);
else{
response.setHeader('content-type', 'image/png');
response.send(content);
}
});
});

app.post("/upload", function (request, response) {
//check cache size
skipperMemory.ls(tempFolder, function(err, fileNames){
if (err){
response.status(500).send(err);
}else{
console.log("files in cache: " + JSON.stringify(fileNames));
async.whilst(
function () { return fileNames.length >= appEnv.cacheSize; },
function (callback) {
//remove old files from cache
skipperMemory.rm(tempFolder + "/" + fileNames[0], function(err, content){
if (err)
callback(err);
else{
fileNames.shift();
console.log("file removed from cache. New cach is: " + JSON.stringify(fileNames));
callback(null, fileNames);
}
});
},
function (err, fileNames) {
if (fileNames)
console.log("files in cache after cleaning oldest: " + JSON.stringify(fileNames));
if (err){
response.status(500).send(err);
}else{
var fileName = new Date().getTime().toString();
console.log("new fileName: " + fileName);
request.file('file').upload({
adapter: require("skipper-memory"),
dirname: tempFolder,
filename: fileName
}, function (err, uploadedFiles) {
if (err) {
console.log(err);
response.status(500).send(err);
}
else {
response.json(uploadedFiles);
}

});
}
}
);
}
});
});

mem_settings.json
=================
{
"port": 6080,
"cacheSize": 10
}

-POSTMAN test queries:

POST http://localhost:6080/upload
request body is form-data with 'file' element of type file. Select a *.jpg image.

GET http://localhost:6080/latest
response is the latest image uploaded to memory cache

[Orit Prince]

Here is a solution for using skipper as httpNodeMiddleware in NR, and extract attached files from HTTP requests with no file system interaction:

-npm install skipper
-npm install skipper-memory
-npm install streamifier

-settings.js:
... httpNodeMiddleware: function(req,res,next) {

skipper(req,res, next);
},
functionGlobalContext: {
skipperMemory: require('skipper-memory'),
streamifier: require('streamifier')
},

-Create new node, named 'skipper in':
Copy the 'HTTP in' standard NR node into NR nodes folder, rename it to be 'skipper-in'.
Remove definition of 'HTTP out' and 'HTTP request' nodes, and leave only 'HTTP in'. Rename node type 'HTTP in' to be 'skipper in'.
Modify the rawBodyParser function as pasted below

function rawBodyParser(req, res, next) {
if (req._body) { return next(); }
//req.body = "";
req._body = true;

var isText = true;
var checkUTF = false;

if (req.headers['content-type']) {
var parsedType = typer.parse(req.headers['content-type'])
if (parsedType.type === "text") {
isText = true;
} else if (parsedType.subtype === "xml" || parsedType.suffix === "xml") {
isText = true;
} else if (parsedType.type !== "application") {
isText = false;
} else if (parsedType.subtype !== "octet-stream") {
checkUTF = true;
}
}

/* getBody(req, {
length: req.headers['content-length'],
encoding: isText ? "utf8" : null
}, function (err, buf) {
if (err) {
console.log("getBody error: " + JSON.stringify(err));
return next(err);
}
if (!isText && checkUTF && isUtf8(buf)) {
buf = buf.toString()
}

req.body = buf;
next();
});*/
next();
}

-Sample flow which uses skipper as httpNodeMiddleware, and skipper-memory adapter for extracting the attached file into 'memory files':

[{"id":"4fb35e35.a568","type":"swagger-doc","z":"7816e80c.06e138","summary":"","description":"","tags":"","consumes":"","produces":"","parameters":[{"name":"task","in":"formData","required":false,"type":"string"},{"name":"image","in":"formData","required":false,"type":"file"}],"responses":{},"deprecated":false},{"id":"60b8e5d2.f1ca4c","type":"function","z":"7816e80c.06e138","name":"read cached image into form data","func":"node.log(\"hi there\");\n\nvar skipperMemory = global.get('skipperMemory')();\nvar streamifier = global.get('streamifier');\n\n//Retrieve attachement from mem cache \nnode.log(\"going to retrieve from mem cache \" + msg.payload.memFile);\nskipperMemory.read(msg.payload.memFile, function(err, content){\n if (err){\n msg.payload = err;\n node.error(\"Error\", msg);\n }\n else{\n node.log(\"retrieved from mem cache: \" + msg.payload.memFile + \" 'file size: \" + content.length);\n formData = msg.payload;\n formData.attachments = streamifier.createReadStream(content);\n formData.attachments.path = \"dummy path which is not used but should exist\"; \n msg.payload = formData;\n node.send(msg);\n }\n});\n","outputs":"1","noerr":0,"x":565,"y":111,"wires":[["1f07e75b.9a9c29"]]},{"id":"a1969e8c.e2c22","type":"function","z":"7816e80c.06e138","name":"cache image","func":"node.log(\"hi there\");\n\nvar skipperMemory = global.get('skipperMemory');\nvar streamifier = global.get('streamifier');\n\nvar tempFolder = \"/temp\";\n\n//Upload attachement to mem storage\nvar memFileName = new Date().getTime().toString();\nnode.log(\"memFileName: \" + memFileName);\n\nvar formData = msg.req.body;\nnode.log(\"msg.req.body: \" + JSON.stringify(formData));\n\nmsg.req.file('image').upload({\n adapter: skipperMemory,\n dirname: tempFolder,\n filename: memFileName\n}, function (err, uploadedFiles) {\n if (err) {\n node.log(\"error while uploading file to mem cache\");\n msg.payload = err;\n\t\tnode.error(\"Error\", msg);\n }\n else {\n node.log(\"uploaded to mem cache \" + JSON.stringify(uploadedFiles));\n if (uploadedFiles.length <= 0){\n return msg;\n }\n formData.memFile = uploadedFiles[0].fd;\n var messages = [];\n msg.payload = formData;\n messages.push(msg);\n messages.push(msg);\n node.send(messages);\n }\n});\n","outputs":"2","noerr":0,"x":298,"y":117,"wires":[["60b8e5d2.f1ca4c"],["4f0ea50b.85fbac"]]},{"id":"1f07e75b.9a9c29","type":"function","z":"7816e80c.06e138","name":"Send form data to REST service","func":"var request = global.get('request');\n\nvar url = \"http://myRESTHost:9090/RESTPath\";\nvar streamifier = global.get('streamifier');\n\nformData = msg.payload;\nformData.ClientID = 'myClient';\n\nvar options = {\n url: url,\n formData: formData,\n 'content-type': 'multipart/form-data'\n};\n\nrequest.post(options, function(err,response,body) {\n if(err){\n node.error(err);\n node.send({payload: err})\n } else {\n node.warn(\"RESOPNSE:\\n\" + JSON.stringify(response));\n msg.payload = JSON.stringify(response); \n node.send(msg)\n }\n});\n","outputs":1,"noerr":0,"x":897,"y":111,"wires":[["f81af2f3.3ef6d"]]},{"id":"56acef0.7bbfc1","type":"skipper in","z":"7816e80c.06e138","name":"","url":"/skipperTest","method":"post","swaggerDoc":"4fb35e35.a568","x":107,"y":117,"wires":[["a1969e8c.e2c22"]]},{"id":"4f0ea50b.85fbac","type":"function","z":"7816e80c.06e138","name":"read cached image into payload","func":"var skipperMemory = global.get('skipperMemory')();\n\n//Retrieve attachement from mem cache \nnode.log(\"going to retrieve from mem cache \" + msg.payload.memFile);\nskipperMemory.read(msg.payload.memFile, function(err, content){\n if (err){\n msg.payload = err;\n node.error(\"Error\", msg);\n }\n else{\n node.log(\"retrieved from mem cache: \" + msg.payload.memFile + \" 'file size: \" + content.length);\n msg.payload = content;\n node.send(msg);\n }\n});\n","outputs":"1","noerr":0,"x":559,"y":281,"wires":[["ff84a1fd.0678f"]]},{"id":"f81af2f3.3ef6d","type":"function","z":"7816e80c.06e138","name":"delete cached image","func":"//image can be removed only if not used anymore by any other node\nvar skipperMemory = global.get('skipperMemory')();\n\n//Delete attachement from mem cache\nskipperMemory.rm(msg.payload, function(err, content){\n if (err){\n msg.payload = err;\n node.error(\"Error\", msg);\n }\n else{\n node.log(msg.payload + \" removed from mem cache\");\n node.send(msg);\n }\n}); \n","outputs":1,"noerr":0,"x":1207,"y":110,"wires":[["c1144386.7915c"]]},{"id":"ff84a1fd.0678f","type":"debug","z":"7816e80c.06e138","name":"","active":true,"console":"false","complete":"false","x":788,"y":424,"wires":[]},{"id":"c1144386.7915c","type":"http response","z":"7816e80c.06e138","name":"","x":1287,"y":468,"wires":[]}]