File upload progress - chunks seem to get bigger, progress slows

[Paul Sanderson]

When using the firebase storage progress the chunks or amount uploaded before a progress event get bigger and bigger, on big files this looks like it is hanging to the user.

Eg. when i upload a 100MB file the progress events fire at these points

0.262144

0.786432

1.835008

3.93216

8.126464

16.515072

33.292288

66.84672

100.401152

complete

Also note that these progress points are exactly the same everytime I upload the same file.

Is there something that I'm missing as I thought the progress was meant to fire more often than that. I looks really broken to the user.

FYI here is my progress code? am i missing something or is there are way to controll the chucking etc

// setup for processing the progress of the upload

    uploadTask.on(firebase.storage.TaskEvent.STATE_CHANGED, (snapshot) => {

      debugger

      let percent:number = Math.floor(snapshot.bytesTransferred / snapshot.totalBytes * 100);

      //get all the files that are being uploaded for this file item

      let uploadingFiles:IUploading[] = this.filesUploading[snapshot.metadata.customMetadata.fileItemId];

      // find the file that we gor the progress for

      uploadingFiles.forEach((v:IUploading) => {

        if(v.meta.customMetadata.uploadRef == snapshot.metadata.customMetadata.uploadRef) {

          v.uploadProgress = percent;

          v.bytesTransferred = snapshot.bytesTransferred;

          v.totalBytes = snapshot.totalBytes;

        }

      });

    });

[Alex Memering]

So behind the scenes the Storage SDK is doubling the amount that is uploaded every time.  This is because it is a resumable upload, and in an attempt to balance speed of the upload with the fact that the network could be lost at any point and that particular portion of the upload could fail the file is broken up into chunks.  The amount uploaded is going to double upon a successful request (this will add a little bit of speedup given that there is a larger amount of file compared to request overhead), and if the request fails for any reason then it will restart at the smallest size again.

There isn't a way to control this behavior and admittedly this makes for a weird progress bar if you're expecting it to grow linearly.  It is something we are aware of and we plan on trying to expose more progress events so that you get 'smoother' updates, but I don't know when that will happen.

Sorry that I don't have a more helpful answer,

Alex

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/6bdeecd0-83d5-41c3-8e65-eb269be2cea9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex Memering |

Software Engineer |

meme...@google.com

[Paul Sanderson]

I understand as that is what I have been seeing during my testing. 

1 more question If it is doubling every time when we are uploading a 100GB at some stage we would be waiting for gigs to upload before getting a progress resp? Is a this correct assumption?

[Alex Memering]

Sorry for the delay.  From my understanding of how it was working, yes, if you were uploading something really massive you might be potentially waiting for very long periods between progress updates.  And that would be super annoying, both from the end-user perspective and developing with it (but to be fair I don't think that most people are going to be uploading files that large).

However there is some good news.  The engineer that originally wrote that part of the SDK fixed that bug, as you can see in the release notes for JS SDK 3.7.5.  I believe that the progress now is reported more on a per request basis now, meaning that in addition to when the chunks themselves complete there will also be notifications as parts of the chunks upload.

I hope that this is more of what you had wanted,

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/e9ba56a4-ee63-4d0f-a163-c38f956533c1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.