upload file to actionjs

[Suman Adak]

Dear,

I want to upload a file through rest api. I would like to have a api like curl -X POST --data-binary @new.zip localhost:8080/upload . How do I consume file in action and save it disk with same file name?   I mean consume file as a body in request. Its just a api not a form upload. please help me..

[Evan Tahler]

Hello! 

First, the act of uploading a file in this manner does actually change your request to a form upload.  You can check your request headers from within in an action (data.connection.rawConnection.req.headers) and see that 'content-type' = 'application/x-www-form-urlencoded' That's just how HTTP works.  

If you upload a file in that manner, ActionHero doesn't really know what to do with it.  You are sending the payload of the upload as the request, and not keyed to any parameter.  Doing it this way, you can't do validation on it (key must exist, be so long or short, be of a certain file type, etc) because it's just coming in as the request body.  The normal way to do this is to send that data up with a parameter key, perhaps something like `data`, ie: curl -X POST -F file=@file.txt localhost:8080/api/upload Then, in your action if you look at data.params.file, you can see all the data you would ever want:

{ file:

   File {

     domain: null,

     _events: {},

     _eventsCount: 0,

     _maxListeners: undefined,

     size: 1148,

     path: '/var/folders/ry/c05q3yxs1q98yps6h2_g3gn00000gn/T/upload_ac2b28e25a253305374c3f44e649fbad',

     name: 'file.txt',

     type: 'text/plain',

     hash: null,

     lastModifiedDate: Sat Jul 09 2016 17:02:40 GMT-0700 (PDT),

     _writeStream:

      WriteStream {} },

  action: 'upload',

  apiVersion: 1 }

If you must upload the file in the way you are describing, you'll have to bypass ActionHero's entire middleware and params chain.  The data you upload will appear as the data.connection.rawConnection.params.body, but you will never know the original files name, as it is not sent with the request...

[Suman Adak]

Dear Evan,

Thanks for reply. I understood. I tried with same and worked fine. But the API does not mention any param name for file. it just attached a zip file as body. 

 It is same like google upload API.

 https://developers.google.com/drive/v3/web/manage-uploads#simple

if i would design my api same as google simple upload api, how would I  do with actionhero?  may be I am misunderstanding something wrong. please help me to understand.

[Evan Tahler]

To get the file name, I belive Google Drive is asking for the filename via URL: The format of the upload endpoint is the standard resource URI with an “/upload” prefix.  Since, as noted, there can be no  metadata (filename) included with a body post like this, the actual URL would be the file name, IE: POST /upload/drive/v3/path/to/my/file.

You can certainly do something that with ActionHero too with creative use of routes. 

[Suman Adak]

Dear Evan, 

Thanks a lot. I got the point. In multi-part , I see an uploaded file is always first written  in temporary location . and then it can be re-written to other file or send to somewhere. Cant we write to memory buffer directly rather than temp file? As I need to send the uploaded file to rabitMQ/ redis for queuing to process.  I was thinking to use task where I will pass the file object reading form memory and save/sent to other channel. writing temp file could be bottleneck for us.  

On Sunday, July 10, 2016 at 11:42:11 PM UTC+5:30, Evan Tahler wrote:

To get the file name, I believe Google Drive is asking for the filename via URL: The format of the upload endpoint is the standard resource URI with an “/upload” prefix.  Since, as noted, there can be no  metadata (filename) included with a body post like this, the actual URL would be the file name, IE: POST /upload/drive/v3/path/to/my/file.

[Evan Tahler]

There are ways to upload files into memory rather than /tmp, but I strongly advise against that, and will never even enable an option for ActionHero to do so.  Here are a few of the reasons:

- You expose yourself to the easiest denial of service attack possible.  I'll just keep POSTING 1 large file ~1GB to you until you run out of RAM and crash

- You can't truncate/terminate the upload mid-stream if you can detect that you are running out of RAM

- Parsing the file is only a sync option now (you can't load the file in like you would fs.readFile), as it is already present in RAM, and doing anything with that data will be blocking

That said, if you are worried about *speed*, there are a few OS-level hacks you can do:

- The location where you save the tmp uploads is configurable.  You can mount a faster, but small hard drive to /mnt/fatDrive/tmp and put the files there

- You could also mount a RAMDISK and use that as the tmp uplaod location.  What's nice about a ram disk is that you'll allocate a fixed space for it, not an ever-growing heap which would be the case if you uploaded files into node's RAM directly. 

All of this said, reading a file from local disk is really fast.  Are you sure that you'll have trouble reading at the throughput you need?  Assuming your uploaders are sending files from outside of your datacenter, the time to upload will be an order of magnitude slower than local disk access, even at high volumes.  Any extra time added from reading a disk will be negligible. 

[Suman Adak]

Dear Evan,

thanks for detail explanation. My requirement is like this, I have to upload a 1kb zip file from 100 million phones for stats in a weekly basis with roughly 500 request per sec. I will be on  restricted network. So I am not expecting any DoS as api will be guarded with some auth key.  Request comes with time distribution within entire week. As I am transferring files to some other system for processing and storing, I wanted to reduce disk space required for uploaded api server. or else I have to have a cron job/task  to clean up disk space once file is send to queue. I was thinking to avoid two write cycle. one for writing temp and another for writing to queue. But you are absolutely correct . In such scenario, I would go like that

1. Write a data to temp/RAMDISK file

2. fire a async task( actionhero task)  with file path which will read data from tmp/RAMDISK and send data to rabbitMQ/redis or processing file and delete the temp file

3. return response to client

I will keep posting benchmark/performance metric over a period of time for actionhero and with nginx/lua based solution too. if anything can be done in better way, please let me know. thanks lot for help. 

thanks

suman

 

  If I could quickly read file from memory and send to queue,

[Evan Tahler]

I think you missed the end of that last sentence.   

Guarding your API with auth key won't prevent a DOS. The *entire* upload needs to be parsed and loaded into RAM before it can be read... to see if that auth key is present.  That's why uploads always go to a file first, so you can throw them away (and not waste RAM) unless you want them loaded in, which you have to do late. 

Your use of the task system seems like a good idea! 

By the way, 1M uploads over 7 days is actually only ~2req/sec if you evenly space it out.

Good luck!  Join us on Slack if you want to talk more!