How to increase file transfer rate

[Anirban Dutta]

Hi,

This is may be a bandwidth question, but let me ask.

I was using datachannel file sharing demo. And i wanted to add progress bar. So, i needed to adjust that bar with file uploading downloading progress. I have some questions:

* Can i increase my file transfer rate through increasing maxPacketSize ? Even with a theoretical bandwidth of 54Mbps between my Tablet and Laptop, I get an avg of 900KBps for small files (around 10-20MB). For large files (153MB) it slowed down to 450KBps average.

* Is there any default file compression algorithm used in easyrtc before passing binary data to RTCPeerConnection ?

* I've to connect to internet to establish a data channel file sharing between two peers, where audio-video works fine withoutt internet. Is that a DNS issue ?

-Anirban

[Eric Davies]

It used to be the case that chrome imposed a limit on how much bandwidth the datachannels could use because they wanted to be sure that it didn't impact the audio/video channels.  That may still be the case.

Easyrtc doesn't do any file compression on data channels themselves, and our file transfer code is very rudimentary (convert to base64, chop into pieces, send pieces using a sliding window protocol so send-buffers don't get too bloated). You could certainly improve on our file transfer code.

However, it sounds like you are behind a packet shaper. Packet shapers are designed to squeeze the bandwidth of connections to reduce network load. Talk to your network admin.

> I've to connect to internet to establish a data channel file sharing between two peers, where audio-video works fine withoutt internet. Is that a DNS issue ?

Data channels usually work wherever audio video works. However, it does use a different network protocol (sctp) than audio video (rtp and rtcp), so it is possible that a network admin could block one but pass the other.

[Anirban Dutta]

Are those chopped parts all equal ? I logged from sending and receiving. It seemed the data was sent every 10240 bytes. The variable maxChunk also has a size of 10240. I tried to change that if it would increase transfer speed for large files, but no result.

Can I bypass the limit imposed by chrome? I am not using audio-video and file transfer together.

[Eric Davies]

All but the last part of a file should be roughly the same size.

See https://code.google.com/p/webrtc/issues/detail?id=4468

[Anirban Dutta]

About establishing connection without internet. without connecting to internet, Audio video communication worked fine before. But now on chrome, it shows only black screen and no error on console. On FF , console shows ICE failed.

RTCIceServer.url is deprecated. Use urls instead.
ICE failed, see about:webrtc for details.

Both from unknown lines.

[Eric Davies]

Sometimes google translate works, sometimes it doesn't. This time it doesn't. 

No idea what your use case is.

[Anirban Dutta]

When I try to initiate a call, performcall() works fine! Other party gets a accept / reject option. When he accepts, we both get black screen. Maybe a Call has been initiated but stream isn't transferred to other side.

[Eric Davies]

ice is failing. you probably need a turn server. if you think you have a turn server, check it's actually running.

[Anirban Dutta]

Thank Eric.
As ICE was failing between two browsers on the same machine, I tried to create a network from my Tablet(without internet access) and connect my server machine to that network. Everything worked fine between chrome on tablet and chrome on server machine.. I'm confused now if it's a turn server issue. 

[Eric Davies]

okay, that's not going to be turn server :-).

on the failing machine, have you denied access to the camera/microphone to that site? Thats the only thing that comes to mind in this case.

[Anirban Dutta]

No. I didn't deny media access. But i noticed a thing, if my server isn't connected to a network, ICE fails. If i connect it to any network(i.e with or without internet access) everything works fine. It's kind of weird.

[Eric Davies]

Okay, that I have seen before. When we're at tradeshows, we normally run connected by a router that isn't tied to the outside world. But one occasion, we only brought one router and a colleague needed to use for a presentation, and voila, no connection. It was an annoying discovery. However, not being able to run webrtc applications when you aren't connected to anything at all isn't what I'd consider a serious use case :-).

[Anirban Dutta]

Yeah.. may be that not serious. But unusual happening, deserves investigation. :P

Back to my topic. Quoting from html5rocks, "The current recommendation for maximum chunk size is 16KB." It seems easyRTC allows 10KB highest. Shouldn't simply increasing values at maxChunksize would increase file transfer a little ?

[Eric Davies]

I just took a closer look at the code. It's been a long time since I wrote it, and at the time chrome could only send relatively short packets.

So, there are three parameters

    maxChunkSize - is actually the amount of data that is read in at a time

   maxPacketSize - is the amount of data that is sent in a single message before base64 encoding. This was set at 400 bytes (miserably small).

   ackThreshold - is how much data can be sent before waiting to receive an ACK from peer we are sending the data to.

The maxPacketSize was buried deep in the code, when it should have been higher up so it could found more easily.

I've just pushed a change to the beta branch that makes these parameters easier to find and modify. The code in easyrtc_ft.js now reads:

   //

    //  maxPacketSize is the size (before base64 encoding) that is sent in a

    //               single data channel message, in bytes.

    //  maxChunkSize is the amount read from a file at a time, in bytes.

    //  ackThreshold is the amount of data that can be sent before an ack is

    //               received from the party we're sending to, bytes.

    //  maxChunkSize should be a multiple of maxPacketSize.

    //  ackThreshold should be several times larger than maxChunkSize. For

    //               network paths that have greater latency, increase

    //               ackThreshold further.

    //

    var maxPacketSize = 40*1024; // max bytes per packet, before base64 encoding

    var maxChunkSize = maxPacketSize * 10; // max binary bytes read at a time.

    var waitingForAck = false;

    var ackThreshold = maxChunkSize * 4; // send is allowed to be 400KB ahead of receiver

Give it a try and let me know if you see faster file transfers.

[Anirban Dutta]

:D It definitely did, 1.5 times the previous transfer rate.

Some questions came:

* Is there any limit on maxPacketSize ? I mean how big chunk dataChannel can handle now? I actually wanted to use maximum of my network.

* About ackThreshold. Maybe a very noob question.
Why do we need it? I wondered why there would be data in 'bitReceivedPerSecond' graph at webrtc-internals for sender. Are those bits of ackThreshold ?

And, are there any such tweaks available for mediaStreams ? Though they don't transfer binary data, but handling resolutions would be great !

[Eric Davies]

max packet size: it doesn't make any sense to send packets larger than the network supports because they'll get broken up anyhow. IPV4 supports 64K packets. Besides, if you are only seeing 50% faster transfer rate, you've already reached the point of diminishing returns. However, feel free to tweak the numbers and see what happens. 

The ack threshold is there so that the sender doesn't send data faster than the receiver can receive it. If you want to know if it is needed, try making it larger than any of your files so that the sender never waits.

But understand, that file transfer code was written in 2 days. It was intended to be a small simple component, not a high performance one. There are more elaborate systems out there that get faster transfer. If speed is important to you, look for them. 

[Anirban Dutta]

Though it was written in 2 days, it's quite elegant for a local setup. :)

I just wanted to test the highest capacity of my setup and see if i could add some compression algorithm to it.

Though I didn't mean production level performance thorough 'high transfer rate', But i appreciate your suggestion.