Firebase Cloud Storage: Slow and unreliable

[Bill Tricarico]

I have been trying to pump 3500+ XML files up to Firebase Cloud Storage.   Since everything is asynchronous, I should just be able to put the upload logic in a for loop, and fire and forget.  When I check for the errors related to the uploading, there are many.  In fact, I would say maybe 20% of the files end up in the Storage, the rest error out on timeouts, socket hangs up, etc.   These are not large XML files, most are event less than 1k in size.  In addition, some files that do get uploaded still report an error, so it's not easy to determine if the file was uploaded or legitimately errored out.

Also, if I use the web page in Firebase to bulk upload a large number of files, it will just hang there.  I cannot tell if it's uploading or not.  At least if I go directly to the Google Cloud Storage page, I can see the upload progress.

Is there a preferred/correct way to bulk upload lots of files to Firebase Cloud Storage?

Thanks for any help that can be provided.

[Kato Richardson]

Hi Bill,

Probably going to be mostly guesswork without seeing some of the specifics (code, qps, file sizes, et al) and the errors you've encountered. You may want to check out best practices and if you're not on the pay-as-you-go plan, double check the daily limits. I suspect the paid plans still have some quota enforcement as well that you could be hitting, depending on frequency of your uploads.

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/15b7aad1-cb6f-4ce8-bf6d-db681d0a4a76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398

[Bill Tricarico]

I am on the pay-as-you-go plan.   The XML files range in size from < 1k to 800k.  My code to upload the XML files is below:

function uploadFile(file) {

    bucket.upload(file, function(err, remoteFile) {

        if (err) {

            console.error(err);
        }
        else {

            // delete local file
            fs.unlink(file, function(err) {

                if (err) {

                    console.log(err);
                }
            });
        }
    });
}

function uploadBulkFiles() {
    
    var files = fs.readdirSync("/xml");
    for (var i = 0; i < files.length; i++) {

        var file = files[i];
        uploadFile(file);
    }

} 

On Wednesday, June 14, 2017 at 1:04:02 PM UTC-7, Kato Richardson wrote:

Hi Bill,

Probably going to be mostly guesswork without seeing some of the specifics (code, qps, file sizes, et al) and the errors you've encountered. You may want to check out best practices and if you're not on the pay-as-you-go plan, double check the daily limits. I suspect the paid plans still have some quota enforcement as well that you could be hitting, depending on frequency of your uploads.

☼, Kato

On Wed, Jun 14, 2017 at 9:55 AM, Bill Tricarico <billtr...@gmail.com> wrote:

I have been trying to pump 3500+ XML files up to Firebase Cloud Storage.   Since everything is asynchronous, I should just be able to put the upload logic in a for loop, and fire and forget.  When I check for the errors related to the uploading, there are many.  In fact, I would say maybe 20% of the files end up in the Storage, the rest error out on timeouts, socket hangs up, etc.   These are not large XML files, most are event less than 1k in size.  In addition, some files that do get uploaded still report an error, so it's not easy to determine if the file was uploaded or legitimately errored out.

Also, if I use the web page in Firebase to bulk upload a large number of files, it will just hang there.  I cannot tell if it's uploading or not.  At least if I go directly to the Google Cloud Storage page, I can see the upload progress.

Is there a preferred/correct way to bulk upload lots of files to Firebase Cloud Storage?

Thanks for any help that can be provided.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/15b7aad1-cb6f-4ce8-bf6d-db681d0a4a76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.