Google Cloud App Engine (GAE) Flexible: Load Balancer error: backend_connection_closed_before_data_sent_to_client
Vit Amin
# [START runtime]
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT ******.wsgi
automatic_scaling:
min_num_instances: 2
max_num_instances: 2
beta_settings:
cloud_sql_instances: ******:europe-west6:*******
runtime_config:
python_version: 3
handlers:
- url: /.*
script: auto
secure: always
redirect_http_response_code: 301
# [END runtime]
After standard instance weekly restarting, I've seen some internal errors in logs and 502 errors on POST requests:
{
"insertId": "2",
"jsonPayload": {...},
"timestamp": "2022-05-14T08:34:42.512018934Z",
"severity": "ERROR",
"logName": "projects/****/logs/compute.googleapis.com%2Fshielded_vm_integrity",
"receiveTimestamp": "2022-05-14T08:34:44.518408254Z"
}
and
{
"insertId": "*******",
"jsonPayload": {
"remoteIp": "******",
"@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
"statusDetails": "backend_connection_closed_before_data_sent_to_client"
},
"httpRequest": {
"requestMethod": "POST",
"requestUrl": "https://*********",
"requestSize": "2208",
"status": 502,
"responseSize": "451",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0",
"remoteIp": "*******",
"referer": "********",
"serverIp": "********"
},
"resource": {
"type": "http_load_balancer",
"labels": {
"url_map_name": "",
"target_proxy_name": "",
"zone": "global",
"project_id": "********",
"backend_service_name": "********",
"forwarding_rule_name": ""
}
},
"timestamp": "2022-05-14T08:36:28.064500Z",
"severity": "WARNING",
"logName": "projects/********/logs/requests",
"trace": "projects/********/traces/********",
"receiveTimestamp": "2022-05-14T08:36:28.882230792Z",
"spanId": "********"
}
Also I have some messages with
/readiness_check failReason:"null"
I've read Tuning NGINX behind Google Cloud Platform HTTP(S) Load Balancer. But as I understand it is related to GCE(not GAE).
For the current app it is VERY important to handle every request, so I want to be confident with fixing it.
Andres Marquez Rodriguez
As per the error "backend_connection_closed_before_data_sent_to_client", the backend unexpectedly closed its connection to the load balancer before the response was proxied to the client. Please, follow this link for more information [1].
The GFE generates this HTTP response code[2] when the connection to the backend is dropped during a request.
It is usually caused because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed (10 minute) keepalive (HTTP idle) timeout of the GFE. There are some situations where the backend may close a connection too soon while the GFE is still sending the HTTP request.
The solution for this issue, is to configure the keepalive timeout parameter for your web server appropriately. (In short, set Keepalive to greater than 10 minutes, such as the recommended value of 620.)
For more information, see the reference Google documentation below in link[3][4].
If it keeps failing after you changed the timeout, I’d suggest you contact Support.
[2] 502s - "backend_connection_closed_before_data_sent_to_client" responses
[3] https://cloud.google.com/load-balancing/docs/https#timeouts_and_retries
[4] https://cloud.google.com/load-balancing/docs/https#statusdetail_http_failure_messages
[5] https://cloud.google.com/support