ATTN: bodyParser changes in 3.4.0

28,369 views
Skip to first unread message

tjholowaychuk

unread,
Sep 7, 2013, 3:43:04 PM9/7/13
to expre...@googlegroups.com

 Express 3.4.0 and Connect 2.9.0 have made some small changes to bodyParser(), and more specifically the multipart() middleware used within it. There
 have been concerns regarding temporary-file usage, however to maintain backwards compatibility for now I've added some documentation on 

  We've also switched to the "multiparty" library, instead of using formidable, which allows you to stream the parts directly to arbitrary 
  destinations without hitting disk. Keep in mind that the destination streams must properly implement node's backpressure mechanisms
  otherwise you're likely to cause large memory bloat causing the process to fail. The "defer" option let's subsequent middleware listen
  on "part" events to stream accordingly instead of writing to disk, providing the convenient req.files object that you might be used to.

  Another alternative if you're concerned is to simply use express.json(), and express.urlencoded(), and leave out multipart() all together. Use
  `if (req.is('multipart/form-data')` and formidable/multiparty/parted directly.

  The tmpfile used is os.tmpDir()'s value, so if you plan on continuing to use disk it's highly recommended to set up a strategy for dealing
  with unnecessary temporary files, this is good practice for any production environment, much like log rotation it is critical to any large
  deployment. An example tool is reap(1) https://github.com/visionmedia/reap. Tools like this should be used regardless of the cleanup technique,
  as application processes may fail at any point in time, and may never have the chance to unlink() the file.

  The default limits for bodyParser(), urlencoded(), multipart(), and json() have also been adjusted. The default limit for multipart is now 100mb,
  and 1mb for the other two. If you anticipate requests larger than this you may pass {  limit: '200mb' } to either bodyParser() or the others. It's
  recommended to use each one individually, bodyParser() is a legacy convenience aggregate of the others, but applying a global .limit option
  between the three of them is not a great choice, as sending 200mb of JSON could halt the application.

  If node sits behind a reverse proxy such as nginx you may easily tweak this behaviour there as well.

  If you have questions, concerns, or suggestions let me know.

Hage Yaapa

unread,
Sep 7, 2013, 3:51:18 PM9/7/13
to expre...@googlegroups.com
The quick response to the concerns is much appreciated, TJ!


--
You received this message because you are subscribed to the Google Groups "Express" group.
To unsubscribe from this group and stop receiving emails from it, send an email to express-js+...@googlegroups.com.
To post to this group, send email to expre...@googlegroups.com.
Visit this group at http://groups.google.com/group/express-js.
For more options, visit https://groups.google.com/groups/opt_out.

Raul Vieira

unread,
Sep 7, 2013, 4:02:58 PM9/7/13
to expre...@googlegroups.com
Thanks.

Sent from my iPhone
--

Joe McCann

unread,
Sep 8, 2013, 8:18:54 AM9/8/13
to expre...@googlegroups.com
Nice work TJ.

Camilo Aguilar

unread,
Sep 10, 2013, 3:39:04 PM9/10/13
to expre...@googlegroups.com
whoa nice work TJ, as always ;D. 


On Sun, Sep 8, 2013 at 8:18 AM, Joe McCann <joseph...@gmail.com> wrote:
Nice work TJ.


--
You received this message because you are subscribed to the Google Groups "Express" group.
To unsubscribe from this group and stop receiving emails from it, send an email to express-js+...@googlegroups.com.
To post to this group, send email to expre...@googlegroups.com.
Visit this group at http://groups.google.com/group/express-js.
For more options, visit https://groups.google.com/groups/opt_out.



--
Camilo Aguilar
Software Engineer


Brian Falk

unread,
Sep 11, 2013, 10:19:43 AM9/11/13
to expre...@googlegroups.com
awesome work!

Aaron Heckmann

unread,
Sep 25, 2013, 9:55:49 AM9/25/13
to expre...@googlegroups.com
I'm curious, since formidable is also configurable to not hit disk, were there any other motivations as well?

Simon Mansfield

unread,
Oct 4, 2013, 8:12:01 AM10/4/13
to expre...@googlegroups.com
I'm experiencing some very odd behaviour regarding this suggestion...

If I use bodyParser() globally, my route that accepts multipart form data works fine (although all routes are susceptible to a "file spam" attack), if I switch to global use of app.use(express.json()) & app.use(express.urlencoded()) and in my route add express.multipart() it fails entirely.

I've stepped into the function returned by express.multipart() and it fails it's first check that req._body isn't null/undefined.

Any idea why this might be happening?

Billy Newman

unread,
Oct 7, 2013, 2:43:00 PM10/7/13
to expre...@googlegroups.com
Any reason why the 'type' property is no longer being set?  Am I to assume that if the Content-Type is set correctly on file upload this should still work as it did in the past?

Thanks,
Billy

Simon Mansfield

unread,
Oct 7, 2013, 5:18:13 PM10/7/13
to expre...@googlegroups.com
Billy, I believe it's due to the change in underlying parser; the content type is now under file.headers I think...?

Sent from Mailbox for iPhone


--
You received this message because you are subscribed to a topic in the Google Groups "Express" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/express-js/iP2VyhkypHo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to express-js+...@googlegroups.com.

Billy Newman

unread,
Oct 7, 2013, 10:29:40 PM10/7/13
to expre...@googlegroups.com
Simon, big thanks you are correct.

file.headers['content-type'] was what I needed.

lucj06

unread,
Oct 2, 2014, 10:18:23 AM10/2/14
to expre...@googlegroups.com
Thanks TJ.
Any way to call an authentication middleware before calling bodyparser to avoid unauthentified client to post huge files ?

greelgorke

unread,
Oct 2, 2014, 10:55:13 AM10/2/14
to expre...@googlegroups.com
put your authentication middleware before the bodyparser. if you need the bodyparser for authentication, then put an alternative before it, or just a custom middleware function.
Reply all
Reply to author
Forward
0 new messages