Slow Cloud SQL access from GAE when compared to Cloud Run

[Jon Middleton]

Is it expected that AppEngine Standard performs significantly slower than Cloud Run when a connection to Cloud SQL (Postgres) is involved?

Testing whether to move away from AWS and which GCP solution to move to and have fired up instances of Cloud Run and AppEngine to test a NestJS API.

Having added logging to isolate just the time taken to query the DB as part of an API call, the same GET request that takes 400ms query time on average in Cloud Run takes 2100ms in AppEngine.

To clarify this is the same source, same build deployed to both and these are the times once warm so I'm excluding the first few requests.

For any code not talking to my Cloud SQL instance, i.e. a dummy javascript method to chew up some CPU time, the performance is more or less identical

Cloud SQL, AppEngine and Cloud Run are all set up in the same region (australia-southeast1), and connection from GAE and Cloud Run to Cloud SQL is for both via Private IP and Serverless VPC.

Since I'm in free tier whilst trialing my options, I tried chat support and basically got down to (verbatim) "cloud run is design to handle data from apis with cloud sql" while "app engine is recommended for webhosting".

I'm assuming this is wrong and and a 3-5x slower query performance to Cloud SQL from GAE isn't an expectation? Any pointers in terms of where this may be failing?

I haven't yet tried via Public IP and wasn't keen to but I will likely repeat my tests on both to see if that makes any difference and post my updates here.

[Amit Sinha]

Hello,

By going through the details you provided and relevant documentation for the settings [1,2], I believe the configuration for both are same. During my research, I am yet to find any reason to believe that App Engine takes more time for the connection. Just wondering, are you able to test it using the public IP? Did you find any connection drop from App Engine in the logs?    

[1] https://cloud.google.com/sql/docs/postgres/connect-app-engine-standard#configuring

[2] https://cloud.google.com/sql/docs/postgres/connect-run#configuring

[Jon Middleton]

As an update - I tried Public IP and same latency issues. It's quite hard to diagnose where this would be but I assume the networking between GAE and Cloud SQL seems hamstrung somewhere along the line, public or private.

I did however bump the app.yaml config up to F4 instead of F1 and that halved the response times and halved the pg times (execution+fetching). I expected resource bumps to improve my apps performance but I didn't expect it to have that much of an impact on querying and fetching for a machine and Cloud SQL instance under no load (executing 1 request every 30 seconds only during testing).

The F4 specs seem to put it on par with what Cloud Run fires up, but the reason I was interested in GAE is I don't think I should require those specs and I'm not talking about a significant amount of data coming back from PG (20-40KB).

So I think my question now shifts to is that expected for GAE? Is there much more of a tight link between CPU and network speeds than I was expecting or am I seeing something strange here? 

At the moment it probably rules GAE out for me and I'd stick with Cloud Run or have to bump up my instance class to B8 to get similar performance.

[Jon Middleton]

Thanks for reply - likewise couldn't find any explanation. I'm going to try Public IP today but from the docs it suggests Private IP should be faster faster, but at least that would rule out any VPC issues that Cloud Run doesn't suffer from.

I'm really not keen to run that configuration in any production capacity but it will be interesting to see the results.

Nothing from logs of value - I logged the queries to ensure both APIs on both endpoints returning the exact same query, and only have a single Cloud SQL DB so they're both getting same row counts. I added OpenTelemetry to the app and the pg plugin is showing slower times. I realized in my first tests as I'm using an ORM the "query" time was pg time + rehydration so wanted to eliminate the rehydration and any CPU bottlenecks being the culprit. There may be a bit of that still at play and Cloud Run scaling better but the times reported by opentelemetry-plugin-pg are double on GAE compared to CloudRun.

Nothing in the logs around dropping, and Cloud SQL load as I'm only in testing is low (2% CPU, no spikes)

On Friday, October 30, 2020 at 5:56:24 AM UTC+11 amit...@google.com wrote:

[Elliott (Cloud Platform Support)]

Thank you for your response. Please let us know what happens.