FileSystem and Reliable mode

[Raúl Jiménez]

Hi all,

This is my first topic, so greetings and very nice project! :)

I'm working on a platform to send large files through WebRTC and I want to use the reliable mode with the FileSystem API (http://www.html5rocks.com/en/tutorials/file/filesystem/) to save the data meanwhile is transferring and when is completed then build the file. Is it possible with the new Reliable mode?

I saw the code on GitHub and I think that I can't do that, it would be great if you could guide me a little bit with some code or an API explanation.

Best,

Raul

[Raúl Jiménez]

Hi all,

Well, I managed to send a file and it seems that it's working because I see through the Chrome console how I'm sending Blobs and receiving ArrayBuffers.

The problem is that I don't know how to manage those ArrayBuffers, is there any listener to get that data?

Here's a few console logs of what I'm sending:

PeerJS:  handleMessage:  ["ack", 0, 6]

PeerJS:  sendWindowedChunks for:  0

PeerJS:  Sending... ["chunk", "0", 6, Blob]

PeerJS:  Sending... ["end", "0", 7]

PeerJS:  Sending... ["chunk", "0", 6, Blob]

PeerJS:  sendWindowedChunks for:  0

PeerJS:  Sending... ["end", "0", 7]

PeerJS:  handleMessage:  ["ack", 0, 7]

PeerJS:  Time:  1087

PeerJS:  Sending... ["chunk", "0", 6, Blob]

PeerJS:  Sending... ["end", "0", 7]

PeerJS:  handleMessage:  ["ack", 0, 7] 

And here's what I'm receiving:

PeerJS:  handleMessage:  ["chunk", "0", 5, ArrayBuffer]

PeerJS:  handleMessage:  ["chunk", "0", 5, ArrayBuffer]

PeerJS:  handleMessage:  ["end", "0", 7] 

PeerJS:  Sending... ["ack", 0, 6] 

PeerJS:  handleMessage:  ["end", "0", 7] 

PeerJS:  handleMessage:  ["chunk", "0", 6, ArrayBuffer]

PeerJS:  Sending... ["ack", 0, 7] 

PeerJS:  handleMessage:  ["chunk", "0", 6, ArrayBuffer]

PeerJS:  handleMessage:  ["end", "0", 7] 

PeerJS:  Complete 0 

PeerJS:  Calling onmessage with complete message 

PeerJS:  Sending... ["ack", 0, 7] 

PeerJS:  handleMessage:  ["end", "0", 7] 

I want to get the ArrayBuffer that have been received in this line: ["chunk", "0", 6, ArrayBuffer]

I hope that anyone can help me.

Thanks in advance.

Best,

Raul

[Michelle Bu]

Hey Raul,

Currently Reliable pieces the file together for you at the end because currently the packets sometimes arrive out of order. If you want, you can add a flag and an extra emit data step to the library when the chunks are received. If you want I can show you where in the code this happens. 

Is this a common use case? If so I will also consider building the library to be more configurable...which may make this library useful even when reliable transfer comes out for streaming use cases like yours. 

Thanks for all your questions! :)

Sent from my iPhone

[Raúl Jiménez]

Hi Michelle,

Thank you for this quick response :)

I'm glad to contribute to this awesome project so if you could guide me a little bit to add that it would be great.

For me, this is a common use case, because I want to save locally with the FileSystem API a large file to avoid crashes due to memory leaks and save the data locally in case the user lost connections, so yes, I think that it could be a good feature to have.

Best,

Raul

[Michelle Bu]

https://github.com/michellebu/reliable/blob/master/lib/reliable.js#L185

This is where a 'chunk' of a file is processed, and the next chunk that is expected is ACKed for.

You would want to emit that chunk if it's sequentially correct--that is, if this if statement passes:

https://github.com/michellebu/reliable/blob/master/lib/reliable.js#L205

Of couse, there's 2 minor issues I see now:

- As an example, if you get chunks 11, 12, 13 while waiting for chunk 10, you have to make sure that those chunks are emitted after you get/emit 10 and before you ACK for chunk 14.

- Since you're emitting chunks, you'd need to come up with a good API to differentiate chunks in one file and chunks in another.

Michelle

[pir...@gmail.com]

Maybe PeerJS/Reliable could emit two new events, one for each received
chunk (independent of the order) and another when new data is received
(ACKed data from in order chunks like in TCP sliding window)?

2013/3/25 Michelle Bu <miche...@berkeley.edu>:

--
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix."
– Linus Tordvals, creador del sistema operativo Linux

[Michelle Bu]

What would the use case be of emitting chunks that are not in-order? There would be double processing on the developer's end that is already done in Reliable--there's no way (that I know of) to construct partial files with the FileSystem API with non-sequential chunks. 

I definitely agree that the in-order chunks should be emitted in some way. What do you think would be a good API for this?

Michelle

[pir...@gmail.com]

> What would the use case be of emitting chunks that are not in-order? There
> would be double processing on the developer's end that is already done in
> Reliable--there's no way (that I know of) to construct partial files with
> the FileSystem API with non-sequential chunks.
>

They would be used when order is not important like in UDP, but it's
necesary that transmision is "reliable" and to be sure that packages
have arrived. Related with downloading file, it doesn't matter the
order in what the chunks are received if you write them in the correct
place on the file, writing it in a random order. Needing to have them
in order you can write the file appending it, but you'll have to wait
to the chunks to be in order. I know, you'll need to know what's the
order, but maybe it would be used a sequence ID like in TCP?

> I definitely agree that the in-order chunks should be emitted in some way.
> What do you think would be a good API for this?
>

The same way Python Twisted works :-) Python Twisted add the data to a
buffer, and when got them in order, it emit a onData event with all
the data in the buffer (and frees it). Later, by default and/or
depending of the protocol being used, it has other "sub-events" for
more procesed data with their own buffers, like onLineReceived when a
linefeed is detected or onHttpRequest.

[Raúl Jiménez]

Hi all,

I think that the buffer method makes sense for me but that should be something transparent when using the API.

I would do a few tests and post the results here.

[Raúl Jiménez]

Hi again,

Michelle, do you have a PeerJS version without the reliable.js and util.js minified inside?

I'm having problems trying to use my own Reliable class and it seems that there's a lot of dependencies in the project. Also, seems that there's an Util object on Reliable and an Util object on PeerJS and if they're not minified they're overriding each other. Could you please publish a PeerJS minified version without Reliable?

Maybe you could create two js files, a peer.min.js and a reliable.min.js, maybe it's clearer for everyone.

Thanks

[Michelle Bu]

Hey Raul, I actually compile reliable into peerjs without its util because it uses the same one as peerjs. You can easily rebuild by: 1. cloning the repo, 2. pulling in the submodules , 3. running mom install -d in your shell, 4. going into bin/build.js and find and remove deps/reliable/lib/reliable.js reference, 5. Remove the if statements in lib/dataconnection.js that branch to using reliable, and finally 6. run node bin/build.js to rebuild the minified version. 

Alternatively you can leave it in and use reliable:false and write your reliable code over that. 

I'm away from a computer until Friday night, but let me know if you need any clarification. I can build it for you when I get back if you still need it then

Sent from my iPhone

[Michelle Bu]

Any updates on this?

- Michelle

[Raúl Jiménez]

Hi Michelle,

Well, I fixed a few errors but new ones have been appeared. Could you please do that build for me? I want to try the changes that I have done to Reliable.js to see if it's working.

Best,

Raul

2013/3/30 Michelle Bu <analogm...@gmail.com>

--
Raúl Jiménez