What's the best way to send chunked files with additional info over reliable RTCDataChannel?

[Szymon Nowak]

Hi,

I'm looking for a way to send large files by reading them in chunks using Blob#slice, sending these chunks using reliable RTCDataChannel and merging them together on the receiving side using FileSystem API to avoid storing the whole file in memory.

What's the best way to serialize these blob chunks, so that I can send along some additional data like total number of chunks, number of the current chunk, message type and message id? Should I e.g. convert this additional data to Uint32Array elements and then append file chunk (blob) converted to ArrayBuffer using FileReader#readAsArrayBuffer?

Cheers,

Szymon

[Sam Erb]

Hi Szymon,
On rtccopy.com I use the following method:

1) Broadcast out file information so other users can request that file and have basic information about (name, size/# of chunks) - 

https://github.com/erbbysam/webRTCCopy/blob/master/client/js/file-io.js#L158 & https://github.com/erbbysam/webRTCCopy/blob/master/client/js/file-io.js#L274

1) Use "readAsArrayBuffer" (like you mentioned above) to read in the chunk as binary data that is about to be sent, only when we are about to send the file - 
https://github.com/erbbysam/webRTCCopy/blob/master/client/js/file-io.js#L631

2) As soon as the chunk is received, write it to the filesystem - 

https://github.com/erbbysam/webRTCCopy/blob/master/client/js/file-io.js#L97

The method used to send chunks is, I believe, commonly referred to as "stop and wait" (send a request for each chunk) which works fine for FF->FF transfers as it's possible to send 1MB+ per chunk. However, FF->Chrome or Chrome->Chrome is much slower using this method as Chrome did not implement any non-standard methods for sending/receiving on the datachannel passed a chunk size of 64k - https://code.google.com/p/webrtc/issues/detail?id=2270#c35 (also, FF->Chrome has to be 16k I think - https://code.google.com/p/webrtc/issues/detail?id=2279#c18 )

This is really less than ideal with Chrome so I'm working on a method now to send multiple chunks per request. I'm running into an issue currently where the filesystem API doesn't seem to like writing out of order (or maybe simultaneously?), it also might just be a js scope issue though :) Once I figure out how to make that work, I'm going to update https://www.rtccopy.com / https://github.com/erbbysam/webRTCCopy/ as well as make file transfers work FF->Chrome/Chrome->FF.

Thanks,
Sam

[Szymon Nowak]

Thank you so much, Sam!

When you send a file chunk, do you send any additional data with it (e.g. chunk number) or does the recipient know what to expect (from the file info sent before) and the file chunk itself is enough? I'm thinking if I actually need to append any data to the file chunk (chunk number and so on) or if it should be enough to set the recipient in the correct state - wait for file info, wait for file data etc.

Thanks for the code for writing to the file system - it would take me ages to figure it out myself. I'm currently using "window.URL.createObjectURL(blob)" which is much, much shorter, but of course requires the whole file to be loaded to memory.

Couldn't sender send these 16KB chunks in blocks (e.g. 64 chunks per block) and wait for the recipient to confirm the whole block instead? Not sure how much faster that would be...

[Sam Erb]

Hi Szymon,
If order is guaranteed (there's a flag set when you create the datachannel), than I don't see a reason to have to store the chunk number alongside the packet.

Also, yup, the sender can send multiple chunks per request as you described. You have to be careful though when writing to files using the filesystem API. I was finally able to debug some new code I'm working on using this method and, in Chrome, I was writing to a file that was in the readyState of "Writing" (ie. 2 simultaneous writes). This should have thrown an error but did not. I'm going to create a simple example out of this and likely file a bug against Chrome.

Thanks,
Sam

[Shachar]

You can also check out sharefest.me, and this article in html5rocks exactly on that issue: http://www.html5rocks.com/en/tutorials/webrtc/datachannels/

[Szymon Nowak]

Thanks Shachar.

I know about HTML5 Rocks article - there's a pretty long comment I made there :) 

sharefest.me is great, but it's much more complicated than I need - if I understand it correctly it uses unreliable channels to build bittorrent-like sharing. I'll take a closer look at it (and Sam's rtccopy.com) during the weekend.