multipart file upload example

[Matt Aimonetti]

I couldn't find a good example of how to do multipart file upload with some extra params, so I wrote one:

http://matt.aimonetti.net/posts/2013/07/01/golang-multipart-file-upload-example/

Feel free to let me know if I missed something or did something wrong.

- Matt

[jona...@titanous.com]

Instead of buffering the file into memory twice, you can leverage io.Pipe and a goroutine to stream the part from disk: http://play.golang.org/p/xfNSyyTd15

Jonathan

[jona...@titanous.com]

Fixed two mistakes that I made (this is still completely untested): http://play.golang.org/p/eEFBMGMNTW

[Matt Aimonetti]

Having to use a goroutine makes the example quite more complex, and withou the goroutine the code just blocks. However, I agree that I should have use `copy` instead of reading the content of the file in memory.

Good point also about using "file/path". I updated the blog post with your suggestions, thanks.

On Wed, Jul 3, 2013 at 7:15 AM, <jona...@titanous.com> wrote:

Fixed two mistakes that I made (this is still completely untested): http://play.golang.org/p/eEFBMGMNTW

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/WIFky_-5qpU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jonathan Rudenberg]

On Jul 4, 2013, at 3:04 PM, Matt Aimonetti <mattai...@gmail.com> wrote:

> Having to use a goroutine makes the example quite more complex, and withou the goroutine the code just blocks.

Sure, it's a bit more complex, but it demonstrates the correct way to do this with streaming IO vs reading the entire file into memory for no good reason.

Also, I noticed another bug:

var bodyContent []byte
// …
resp.Body.Read(bodyContent)

bodyContent is a nil slice, so the Read call will read zero bytes. Instead, you can do this:

buf := &bytes.Buffer{}
_, err := buf.ReadFrom(resp.Body)

[Matt Aimonetti]

oh man, how did I miss this huge bug o_O  

Thanks, I updated the blog post ince again.

I added a link to your example with the goroutine, and while it's probably the best approach it does add more complexity.

Can you help me better understand the difference between using a pipe and copying the memory?

It's probably a stupid question but I'm not quite sure I understand the problem. Are you saying that allocation wise, using copy vs piping is the same thing, but the difference is that using copy requires that we read the entire source in memory instead of reading by small chunks? (which could be a problem in case of big files)

If that's the case, wouldn't something like bufio helps?

Again, I realize it might sound like a stupid question but I would like to make sure I fully understand the problem.

Thanks

[Jonathan Rudenberg]

On Jul 4, 2013, at 3:48 PM, Matt Aimonetti <mattai...@gmail.com> wrote:

> Are you saying that allocation wise, using copy vs piping is the same thing, but the difference is that using copy requires that we read the entire source in memory instead of reading by small chunks? (which could be a problem in case of big files)

You are copying the entire file from disk into an in-memory buffer (the bytes.Buffer that you are encoding the multipart request body into). This will fail when the file is too large to fit into memory and is also slower, because you have to wait to read the entire file into memory.

The pipe that my implementation uses never stores more than the chunk size used in io.Copy in memory, streaming it from the disk through to the socket one chunk at a time.

> If that's the case, wouldn't something like bufio helps?

bufio exists to solve a different problem by combining lots of tiny read/write syscalls and providing helpers for some standard operations like reading a line at a time.