AppEngine crashing my webserice randomly with errors

148 views
Skip to first unread message

Robert Bartram

unread,
Apr 21, 2022, 10:29:59 AM4/21/22
to Google App Engine
Were getting random AppEngine instance crashing ever so many days. It seems this year to happen allot more then usual.

We get these shutdown errors when it happens in the log console. Any ideas on why this happens?  Is this Google updating the Linux kernel os behind the scenes?  Any way to stop that or set updates only on off hours?

{
  "insertId": "0",
  "jsonPayload": {
    "@type": "type.googleapis.com/cloud_integrity.IntegrityEvent",
    "shutdownEvent": {},
    "bootCounter": "25"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "instance_id": "<hidden>",
      "zone": "us-central1-c",
      "project_id": "<hidden>"
    }
  },
  "timestamp": "2022-04-20T21:07:52.780477835Z",
  "severity": "NOTICE",
  "logName": "projects/<hidden>/logs/compute.googleapis.com%2Fshielded_vm_integrity",
  "receiveTimestamp": "2022-04-20T21:07:54.811246896Z"
}

Darrell (Cloud Platform Support)

unread,
Apr 21, 2022, 7:37:00 PM4/21/22
to Google App Engine
Hi,

Are there any other relevant error logs? I am asking because there are a couple of reasons for the shutdown process to occur[1]. I suggest verifying if your instance is exceeding the maximum memory and checking if there are any CPU spikes prior to the crashing.

I hope this helps!

[1] https://cloud.google.com/appengine/docs/standard/go/how-instances-are-managed#shutdown

Robert Bartram

unread,
Apr 23, 2022, 6:24:25 PM4/23/22
to Google App Engine
Hello Darrell.  

No other error's at all in the logs.  The AppEngine stays at about 650MB consistently and the CPU at 5% or less.  Were not sure why it happens unless Google is pushing out Linus OS updates on the instances forcibly in our day times. We don't know where to look next.

We have our app.yaml set to this so we should be ok.

resources:
  cpu: 2
  memory_gb: 2.0
  disk_size_gb: 10

Amit Sinha

unread,
Apr 25, 2022, 3:19:00 PM4/25/22
to Google App Engine
Hi,

Thank you for sharing this. As it is not showing any other errors in the logs, it could be hard to tell the root cause by looking into this. 

So at this point, I would recommend to report this in the issue tracker by clicking here to investigate it further. Please provide as much information as possible. Also when turning this issue in private you can share the project ID and App Engine version name in the issue tracker link to check further. I hope it helps.

Reply all
Reply to author
Forward
0 new messages