Google App Engine Instance is abruptly shutting down on Standard Environment

[vidhi thakrar]

We are using Google App Engine to ingest large amount of data to Google Cloud FireStore with below configuration:

Basic scaling instance_class: B4

basic_scaling: instances: 1

The overall data ingestion 0f 20GB takes around 1.5 hours. But we have noticed that some time after an hour, instance is abruptly shutting down with below error:

Container terminated on signal 9.

As per this documentation, basic scaling can serve he request up to 24 hours.

We can not see any more details in the logs as well. Also checked the memory usage, B4 has 1024 MB and the app is only utilising up to 700 MB.

If anyone has faced this kind of error, your input would be valuable!

[George (Cloud Platform Support)]

Instance re-starting is normal in the present case, and your application should take this into consideration. Hardware and software failures that cause early termination or frequent restarts can occur without prior warning and can take considerable time to resolve; thus, you should construct your application in a way that tolerates these failures. You may find related detail on the "How Instances are Managed" documentation page.  Some good strategies for avoiding downtime are listed on that page. You may also consider changing your app.yaml with line basic_scaling: instances: 2, or more than 1. 

[vidhi thakrar]

Thank you George for the response. 

In our case, There is a single endpoint which reads a large file from amazon s3 line by line and then ingests a batch of 50 lines as 50 documents on cloud firestore. It roughly takes around 1.5 hours to insert all the data from a file to cloud firestore. Given that instance uptime is not guaranteed, is compute engine a good solution to perform long running task? Or is there any other option you can recommend?

On the flex environment, we tried to first download the whole file on a disk and then start reading line by line from it which has worked for us and not observed abrupt termination. But wanted to know an opinion around what should be the best option to perform long running task on google cloud platform.