[go-nuts] implementing multipart/form-data for http server

[Michael Hoisie]

A while back I implemented this feature in my web framework, and I'd
like to add it to the http package. The implementation in web.go has
been tested with all browsers for the past couple months, and it works
well. Adding it to http will also let me delete some code from
web.go :)

Here's the way I did it (you can also see the code at
http://github.com/hoisie/web.go/blob/master/request.go )

1) create a new type filedata:

type filedata struct {
Filename string
Data []byte
}

2) Add a field 'Files' to request, which has type []filedata. This is
initially empty, but when request.ParseForm is called and the request
contains multipart/form-data, it gets filled in.

3) The files are read into memory all at once.

4) In web.go there isn't a limit to the file size, but it would be
smart to add a variable like http.MaxFileUploadSize

Sound reasonable?

[abiosoft]

Thanks. its been frustrating having difficulties uploading files.

[Andrew Gerrand]

On 1 May 2010 05:56, Michael Hoisie <hoi...@gmail.com> wrote:
> A while back I implemented this feature in my web framework, and I'd
> like to add it to the http package. The implementation in web.go has
> been tested with all browsers for the past couple months, and it works
> well. Adding it to http will also let me delete some code from
> web.go :)
>
> Here's the way I did it (you can also see the code at
> http://github.com/hoisie/web.go/blob/master/request.go )
>
> 1) create a new type filedata:
>
> type filedata struct {
>    Filename string
>    Data     []byte
> }
>
> 2) Add a field 'Files' to request, which has type []filedata. This is
> initially empty, but when request.ParseForm is called and the request
> contains multipart/form-data, it gets filled in.
>
> 3) The files are read into memory all at once.

I understand the implementation reasons for this, but is there a
better way? How is the multipart data encoded in an HTTP request? It
would be great if we could make Request.Files a []Reader.

[Michael Hoisie]

I suppose it could be a reader, but I can't think of a way to be smart
about reading the data (i.e reading a specific part without reading
the rest of the parts). If the reader would be an interface to a byte
array, than why not just provide the byte array?

If there's a hard-coded limit to file size, the server could just
throw a 413 error if people try uploading a massive file. It could be
small by default (like 50 MB).

- Mike

On May 2, 6:10 pm, Andrew Gerrand <a...@golang.org> wrote:
> On 1 May 2010 05:56, Michael Hoisie <hoi...@gmail.com> wrote:
>
>
>
>
>
> > A while back I implemented this feature in my web framework, and I'd
> > like to add it to the http package. The implementation in web.go has
> > been tested with all browsers for the past couple months, and it works
> > well. Adding it to http will also let me delete some code from
> > web.go :)
>
> > Here's the way I did it (you can also see the code at
> >http://github.com/hoisie/web.go/blob/master/request.go)
>
> > 1) create a new type filedata:
>
> > type filedata struct {
> >    Filename string
> >    Data     []byte
> > }
>
> > 2) Add a field 'Files' to request, which has type []filedata. This is
> > initially empty, but when request.ParseForm is called and the request

> > containsmultipart/form-data, it gets filled in.

>
> > 3) The files are read into memory all at once.
>
> I understand the implementation reasons for this, but is there a

> better way? How is themultipartdata encoded in an HTTP request? It

[Russ Cox]

On Mon, May 3, 2010 at 10:17, Michael Hoisie <hoi...@gmail.com> wrote:
> I suppose it could be a reader, but I can't think of a way to be smart
> about reading the data (i.e reading a specific  part without reading
> the rest of the parts). If the reader would be an interface to a byte
> array, than why not just provide the byte array?

Because the interface to the byte array could be stored on disk
if the upload was large?

Russ

[Michael Hoisie]

Ah, that makes sense.

What about this: if the files are greater than some threshold (either
some constant, like 1 MB, or some percentage of
http.MaxFileUploadSize), store it as a temp file?

There's also some other fields that may be relevant with the file
info:

type Filedata struct {
Name string
Data io.Reader,
Size int64,
MimeType string,
}

- Mike

[Russ Cox]

> What about this: if the files are greater than some threshold (either
> some constant, like 1 MB, or some percentage of
> http.MaxFileUploadSize), store it as a temp file?
>
> There's also some other fields that may be relevant with the file
> info:
>
> type Filedata struct {
>    Name string
>    Data     io.Reader,
>    Size int64,
>    MimeType string,
> }

Sure, something like that seems good.
The Data probably shouldn't be a struct field, because
it means you can only read the data once.
Having a separate Open() io.ReadCloser method would
allow multiple passes through the data.

Maybe just call it a File instead of Filedata.

Thanks.
Russ