Request for comment: removing old multipart uploads

[James Reeves]

Hi all,

I've recently been working on a refactor of how multipart file uploads
are handled. This refactor allows you to specify a custom storage
function, so you can store file uploads on different systems. For
example, you might want to have uploaded files sent directly to
Amazon's S3 service.

The problem is what to do with uploaded files we don't want to keep.
Some storage functions will store uploaded in memory, and therefore
will be garbage collected automatically. But other storage functions
will store uploaded files in persistent databases, or on the
file-system.

Currently Ring stores uploaded files in temporary files, which are
then garbage-collected by a background thread. But not all
applications will want this particular behaviour; for instance, Google
App Engine won't allow you to keep a background thread around like
this. Other cloud servers may have similar restrictions.

My feeling is that there's no one right way to handle this; it depends
on your server architecture, and what storage function you want to
use. I'm therefore thinking that cleaning up old uploaded files should
not be the responsibility of the wrap-multipart-params middleware.

Thoughts?

- James

[László Török]

Hi James,

I have to second your opinion, there is no single right way. I believe, as always, provide a sensible default that work in a high % of all cases (if there is one) and the code should give a meaningful error if it has issues in a particular environment.

Just my 2c.

Las

2011/3/23 James Reeves <jre...@weavejester.com>

[James Reeves]

On 23 March 2011 16:22, László Török <ltor...@gmail.com> wrote:
> provide a sensible default that work in a high % of all cases (if
> there is one)

If anyone can think of a sensible default, I'm all ears :)

- James

[László Török]

the one with temp files sounds good to me ( even if it doesn't work on GAE )

2011/3/23 James Reeves <jre...@weavejester.com>

[James Reeves]

On 23 March 2011 16:51, László Török <ltor...@gmail.com> wrote:
> the one with temp files sounds good to me ( even if it doesn't work on GAE )

I was thinking about providing a function to remove temporary files
that have been unmodified for a certain length of time. This could
then be incorporated into a background thread by the user. e.g.

(use '[ring.middleware.multipart-params.temp-file :only (delete-old-files)])

(future
(while true
(delete-old-files (* 60 60 24))
(Thread/sleep (* 1000 60 5)))))

But this doesn't seem like the sort of thing that should be
incorporated into the multipart middleware by default. Or should it?

- James

[James Reeves]

This thread is ancient, and not about multipart request size. You'd be better off in future opening a new thread, rather than adding your question to an unrelated old thread.

Instead of the :content-length key, look for the "content-length" header in the request map. You can also use the ring.util.request/content-length function, which will take the header and parse it into an integer for you. This is effectively the same as the deprecated :content-length key.

Multipart requests may have a content-length, but they can also be streamed. You can either ignore requests without a content-length, throwing the user an error (which to my understanding is not uncommon), or you can accept streaming uploads and limit the input stream to a specific length. The BoundedInputStream function in Apache Commons IO wraps existing input streams and provides a hard stopping point.

- James

On 6 July 2016 at 06:17, Ikuru Kanuma <kanu...@gmail.com> wrote:

Hi James,

I stumbled upon this thread while searching through the web for a solution to reject large requests in ring.
AFIK, this has not yet been implemented in ring core (looking https://github.com/ring-clojure/ring/pull/98) and I am on my own to implement it.

I have tried so far on immutant 2:
- Pulling out the InputStream from the request and inspecting how large the content is. (.available method and trying to read bytes), which returns -1.
- :content-length from the request map, which on the wiki is mentioned to be deprecated.

I understand that the behavior of the internal InputStream is up to the implementation the underlying web server, but
would greatly appreciate it if you could shed some light on the topic (pointers, advise, refirects...)

Regards,

Ikuru

P.S. Your work inspires me all of the time!

[Ikuru Kanuma]

[Ikuru Kanuma]

Thanks for the precise response and sorry for digging up an old thread!

Much appreciated!

Ikuru