Necessary to split webservice due to the request timeout limitations?

[Jack Lee]

I'm currently implementing a (web)service in AppEngine Flexible. Within a single call to the service it will retrieve data from our backend, BigQuery, pass the data to external APIs, wait for their responses for data processing and then respond to the caller with a processing status. With our current implementation it may take well over a minute to complete the request. The service is written in .NET Core and I'm using the Google Cloud .NET libraries.

I understand that AppEngine has some hard limitations such as a maximum time limit of 60 seconds for a HTTP request to an AppEngine [1], and apparently, there is a similar limit for Google Cloud SQL database requests [2]. I suppose the maximum time limit apply for both AppEngine Standard and Flexible environments? Does the timeout limit apply for BigQuery too? Also, according to this StackOverflow Q&A [3] it appears that the hard limit can be overcome by choosing an appropiate instance class but this seems to be a setting for the Standard environment – is that correctly understood?Instance classes are only described in the About article [4] for Standard environment.

If so, what are my architectural alternatives? Right now I can only think of variants of separation of the webservice receiving requests from the business logic that does the data processing and takes time. In other words, a webservice as the frontend which responds immediately with a jobId of some sort and then the client has to queue for the status. In the meantime another process is waiting for new requests and will change the job status when it's picking up a request.

Google Cloud seems to have several frameworks for this: Pub / Sub message queueing, Task Queue, and perhaps thirdly, a cronjob that would frequently read new requests from some database table. I'm still in the process of understanding the difference between Pub/Sub and Task Queue though there doesn't seem to be an .NET API for the latter. But how is the worker service "awakened" when a new message arrives to Pub / Sub? Will the framwork itself take care of that? And will the worker automatically have its request timeout set to 10 minutes instead of the 60 secs for frontend webservices or is it something one configures in the .yaml-file?

I'm getting a feeling - after having googled enough - that going through the bookshelf tutorial in [5] could answer some of my questions. But I could use some advice on which direction to go to avoid the pitfalls. :)

[1]: Dealing with DeadlineExceededErrors  |  App Engine Documentation  |  Google Cloud Platform
https://cloud.google.com/appengine/articles/deadlineexceedederrors

[2]: Pricing and Access Limits  |  App Engine standard environment for Java  |  Google Cloud Platform
https://cloud.google.com/appengine/docs/standard/java/cloud-sql/pricing-access-limits

[3]: php - How to increase Google App Engine request timer. Default is 60 sec - Stack Overflow
https://stackoverflow.com/questions/43115412/how-to-increase-google-app-engine-request-timer-default-is-60-sec

[4]: The App Engine Standard Environment  |  App Engine Documentation  |  Google Cloud Platform
https://cloud.google.com/appengine/docs/standard/#instance_classes

[5]: Using Cloud Pub/Sub with .NET  |  .NET  |  Google Cloud Platform
https://cloud.google.com/dotnet/docs/getting-started/using-pub-sub

[Yannick (Cloud Platform Support)]

Hello Jack, in the App Engine Standard environment it is indeed good practice to have requests that take a significant amount of time to return be handled by the Task Queue or by backend instances. In the case of the App Engine Flexible environment the response time is of one hour.

It is important that you disregard documentation that pertains to the Standard environment if you are developing for the Flexible environment, this includes your link #2 about Cloud SQL.

Regarding the Task Queue and Cron jobs, those are two separate ways to schedule work for App Engine instances. Cloud Pub/Sub is a general purpose tool that allow you to distribute messages to any number of subscribers. Push subscribers are programs running on any computing platform that subscribe to topics and automatically have messages pushed to them, while pull subscribers query Pub/Sub for any messages they can work on. Both of these are essentially always running in order to do their work. There are ways to have dormant computing capabilities consuming no resources until a Pub/Sub message "awakens them" using products like Cloud Functions and Cloud Dataflow.

I hope this helped clear up some confusion! Do let me know if you have follow-up questions.