Handling HTTP File Uploads

[Andrew Gerrand]

This is a proposal for enhancements to the http package with an aim to
make handling file uploads easier.

First, some context. An HTML form like this

<form action="/" method="POST" enctype="multipart/form-data">
<input type="hidden" name="fielda" value="foo">
<input type="file" name="filea">
<input type="hidden" name="fieldb" value="bar">
<input type="file" name="fileb">
<input type="submit" value="Upload">
</form>

will be sent as an HTTP request with a Content-Type of
multipart/form-data, and a request body that contains each field's
data as a separate mime part. The parts each have headers like this
for a normal field:

Content-Disposition: form-data; name="fielda"

Or for a file field:

Content-Disposition: form-data; name="filea"; filename="filename.ext"
Content-Type: image/png

At the moment it's not that hard to handle requests like this. Just
call the MultipartReader method on the *http.Request, and step through
each part one by one. A downside to this is that it's quite verbose,
and doesn't support random/repeated field access nor parsing of the
Content-Disposition header (to get the filename, for example).

My proposal:

Load the entire multipart/form-data body into memory when
http.Request.ParseForm is called. This makes multipart-encoded form
fields accessible via http.FormValue.

Add a http.File struct that looks something like this:

type File struct {
Filename string
ContentType string
Bytes []byte
}

Add a "File map[string][]*File" field to the http.Request. ParseForm
should populate this slice when it encounters non-text fields in a
multipart-encoded request.

Optionally, add a convenience method to Request << FormFile(fieldname
string) ? >> that returns the first *File in the Request.File map,
akin to the FormValue method.

A caveat: This works for most use-cases, but is undesirable when it's
more appropriate to stream the request to disk. In that case, you may
use the existing MultipartReader method and step through the parts
manually. We may want to put a limit on the size of a File attachment
to prevent issues when very large files are uploaded. There are
already ways of causing http servers to allocate arbitrarily large
blocks of memory, though, so this is not a new issue.

Comments appreciated.

Andrew

[Russ Cox]

This proposal seems to assume you can keep the
uploaded file in memory. Part of the design criteria
for the current solution was not to do that.
Large uploads should not be impossible.

Russ

[Russ Cox]

That is to say, I don't think you give enough credence
to the large file upload use case. Changing the API
makes something currently possible impossible.
I think you can make things more convenient without
giving that up.

Russ

[Brad Fitzpatrick]

I'm hesitant to make this too easy for people.

They should at least know what's going on and have to spell out that they either want slurp-to-memory (with a max size) or spool to temp files (perhaps with a temp directory and various policy).

But making it magically part of ParseForm() feels too dangerous.

[Brad Fitzpatrick]

Actually what I describe below could be implemented without any change to the core http library.

You could make a helper that takes an http request and gives you an object (that you should Close() when done) that gives you a map[string][]io.ReadSeeker with some in-memory and some spooled to temp files depending on threshold.  And then convenience methods to like http.Header in the common case of a map key only having one value and the common case of some values being small and wanting a []byte or string.

[Andrew Gerrand]

To do this, don't call FormValue or ParseForm and use MultipartReader.

[Russ Cox]

I'm still worried. To me, ParseForm does not mean
"copy all the uploaded files into memory".

[Andrew Gerrand]

On 7 April 2011 13:26, Brad Fitzpatrick <brad...@golang.org> wrote:
> Actually what I describe below could be implemented without any change to
> the core http library.
> You could make a helper that takes an http request and gives you an object
> (that you should Close() when done) that gives you a
> map[string][]io.ReadSeeker with some in-memory and some spooled to temp
> files depending on threshold.  And then convenience methods to like
> http.Header in the common case of a map key only having one value and the
> common case of some values being small and wanting a []byte or string.

This is similar to another idea I was tossing around: add a FormReader
type to the multipart package that does pretty-much exactly what you
describe.

The in-memory-or-temp-file-on-threshold approach could also be used in
my existing proposal. The key benefit to having as part of the http
package is that FormValue would continue to work as per non-multipart
forms.

Andrew

[Andrew Gerrand]

Revised additions/changes to the http package:

var DiskCacheThreshold = 10e6
var DiskCachePath = "" // defaults to system temp directory

type File struct {
Filename string
ContentType string
}

func (f *File) Read([]byte) (int, os.Error)
func (f *File) Seek(int64, int) (int64, os.Error)

type Request struct {
// existing fields

File map[string][]*File
}

func (r *Request) FormFile(name string) *File

This does add a lot of mechanism to http. Perhaps the heavy lifting
could be done in another package (eg multipart), with this being
simply a veneer.

Andrew