Strange "GCP Storage Egress" billing, can anyone explain ?

[Patrice B]

Hello,

We have an App Engine application that is hosted in GCP Europe (Frankfurt) and with most users located in EUrope.

We sometimes observe periods of strong "GCP Storage Egress between Northern America and Europe" on our billing breakdown.   For example in May 2020, there was none from May 1st to May 19th, then ~30 USD / day from May 20th to May 28th, then none again.   And that happens on and off every month or so, not in a very regular pattern.

Our application exists on GAE since 2015, and up until Sept 2021 we had hardly seen any of this Egress thing at all.   Attached is the profile of this SKU in the last 12 months.

Does anyone know exactly what causes "GCP Storage Egress" ?  When an end-user located in the US accesses some publicly available media in a bucket that is in a GCP datacenter in Europe, is this Egress traffic ?   Or would it be a GAE application in the US reading from the bucket ?    The first case (access to publicly available objects) might exist but would be very limited, and it would not explain the cost.   In the last 12 months, Egress traffic from NA is over 20 000 GB, that really cannot be explained by the few US users that we might have.

Thanks.

[cristianrm]

As stated in the official documentation:

    Egress represents data sent from Cloud Storage in HTTP responses. Data or metadata read from a Cloud Storage bucket is an example of egress.

    Network usage charges apply for egress and are divided into the following cases:

• Network egress within Google Cloud, when egress is to other Cloud Storage buckets or to Google Cloud services.

• Specialty network services, when egress uses certain Google Cloud network products.

• General network usage, when egress is out of Google Cloud or between continents.

As said in this Stack Overflow answer, you could activate Cloud Storage Data Access Logs for investigation purposes. It's deactivated by default because the volume of logs can be huge. 

    Important: Data Access audit logs volume can be large. Enabling Data Access logs might result in your Cloud project being charged for the additional logs usage. For pricing information, see Google Cloud's operations suite pricing: Cloud Logging.

For further assistance, I suggest contacting cloud support by clicking the link at the Google Cloud support page.

[Patrice B]

Thank you for this info.  I had seen the SO answer and had activated Access Logs already, actually, but I have difficulties finding any answer, or clue, within the logs as to what is deemed Egress and what is not.   More precisely, I don't find information in the Access Logs that would help identify Egress traffic, like the origin being North-America.   Nor could I find the actual volume of data for each access, which would allow me to check the total volume.

[Patrice B]

As an example, let me focus on a single record picked randomly within the Data Access Log.   It relates to the creation of a single file object within a (precisely identified) EU bucket, and the Data Access Log indicates an IP of "2600:1900:2000:25:400::".  When entered into an IP locator, this IP address is said to belong to Google LLC and located in Mountain View, California, USA.   So I'm thinking maybe it is this kind of access that is deemed "Egress NA to EU".   However, if I further analyze this single file creation, I find it is performed by our own application (because I can trace it back to a Request Log record), running in Frankfurt (EW-3).  So I'm left wondering if the attribution of trafic to NA (and the related cost) is reliable.   I think if Egress Trafic implies additional costs, then it ought to be clearly identified within the Data Access Log.  

[cristianrm]

As shown on this Stack Overflow answer, from another user having unexpected GCP Storage egress between NA and EU:

    “My theory is that this is due to container images being downloaded from gcr.io (NOT eu.gcr.io) as part of the process of deploying an App Engine version (It says here that gcr.io is currently in the US). I find some evidence of this in the Cloud Build history: there, I see e.g. Pulling image: gcr.io/gae-runtimes/crane:current.” 

This theory could explain what is happening.

Also, as you have activated Google Cloud Storage Access & Storage logs, you can analyze them by importing into Big Query as shown here.

However, I strongly recommend you to contact the Cloud Support team by clicking the link, to have a detailed investigation of your particular case.