Spotty connection between cloud functions and Firebase Storage

[Bartholomew Furrow]

I have a cloud function triggered as follows:

functions.storage.bucket(stagingBucketName).object().onChange

The files uploaded there are consistently close to 2MB in size. My function downloads those files from a bucket called the staging bucket, manipulates them, and then uploads the two resulting files (one of which is identical to the original 2MB file, the other of which is about 20kB) to a bucket called the serving bucket.

This function fails often enough that I've turned on retries and turned up runtime in an effort to see what happens. I've attached a sample set of logs. In this:

- Over the course of an hour, the function tries 20 times. Each time is up to 3 minutes long.

- The download fails 9 times. Failure is a connection error, taking 120 seconds. The eleven times it succeeds, it consistently takes 4 seconds from the start of the function to the end of the download, except for once it takes 0.2 seconds (!?).

- The upload fails 10 times. Failure happens with my function timing out, taking about 170 seconds. When it finally succeeds, it takes about 0.7 seconds.

This isn't unique to this particular file; this function has been triggered independently many times, and failed many times.

I'm assuming this isn't expected behaviour. If anyone on the Firebase team wants to dig into this, the project is photos-for-grandma, and the function is actually called checkForPhotoUpload. The buckets are photos-for-grandma-photo-staging and photos-for-grandma-photos. I'm interested in:

1. Why does the download fail so often?

2. Why does the upload fail so often?

3. I might get back to you about the download time, but I'm not yet certain how much of the 4 seconds is download.

Cheers,

Bartholomew

[Bartholomew Furrow]

Updates on each of the issues. TL;DR: #1 and #2 are both significant issues, but my hypothesis about #1 was wrong.

1. "Connection error" doesn't seem to mean the download is failing; I think it means the function-running process isn't coming online. The evidence is that a logging statement on the first line of my function doesn't log.

2. This is still a significant issue, and happens like 5-40% of the time with files around 2MB (I have small statistics, so my error bars are a bit wide). It seems to happen rarely if at all for files around 100kB.

3. Download time is fine after all; the 4 seconds was coming from somewhere else.

[Bartholomew Furrow]

Is anyone on the team already looking at this? If I haven't heard so in the next day or two, I'll file it as a bug.

Cheers!

Bartholomew

[Doug Stevenson]

Bartholomew,

Definitely file a bug report for this with all the information you gave here.  That will be very helpful.

https://firebase.google.com/support/contact/bugs-features/

Doug