Google Cloud App Engine (GAE) Flexible: Load Balancer error: backend_connection_closed_before_data_sent_to_client

845 views
Skip to first unread message

Vit Amin

unread,
May 16, 2022, 3:06:47 AM5/16/22
to Google App Engine
I have GAE flexible this app.yaml:

    # [START runtime]
    runtime: python
    env: flex
    entrypoint: gunicorn -b :$PORT ******.wsgi
   
    automatic_scaling:
      min_num_instances: 2
      max_num_instances: 2
   
    beta_settings:
        cloud_sql_instances: ******:europe-west6:*******
   
    runtime_config:
      python_version: 3
   
    handlers:
    - url: /.*
      script: auto
      secure: always
      redirect_http_response_code: 301
    # [END runtime]


Screenshot 2022-05-14 130338.png
After standard instance weekly restarting, I've seen some internal errors in logs and 502 errors on POST requests:

    {
      "insertId": "2",
      "jsonPayload": {...},
      "timestamp": "2022-05-14T08:34:42.512018934Z",
      "severity": "ERROR",
      "logName": "projects/****/logs/
compute.googleapis.com%2Fshielded_vm_integrity",
      "receiveTimestamp": "2022-05-14T08:34:44.518408254Z"
    }


and

    {
      "insertId": "*******",
      "jsonPayload": {
        "remoteIp": "******",
        "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
        "statusDetails"
: "backend_connection_closed_before_data_sent_to_client"
      },
      "httpRequest": {
        "requestMethod": "POST",
        "requestUrl": "https://*********",
        "requestSize": "2208",
        "status": 502,
        "responseSize": "451",
        "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0",
        "remoteIp": "*******",
        "referer": "********",
        "serverIp": "********"
      },
      "resource": {
        "type": "http_load_balancer",
        "labels": {
          "url_map_name": "",
          "target_proxy_name": "",
          "zone": "global",
          "project_id": "********",
          "backend_service_name": "********",
          "forwarding_rule_name": ""
        }
      },
      "timestamp": "2022-05-14T08:36:28.064500Z",
      "severity": "WARNING",
      "logName": "projects/********/logs/requests",
      "trace": "projects/********/traces/********",
      "receiveTimestamp": "2022-05-14T08:36:28.882230792Z",
      "spanId": "********"
    }

Also I have some messages with

    /readiness_check failReason:"null"

I've read Tuning NGINX behind Google Cloud Platform HTTP(S) Load Balancer. But as I understand it is related to GCE(not GAE).

 For the current app it is VERY important to handle every request, so I want to be confident with fixing it.

Andres Marquez Rodriguez

unread,
May 18, 2022, 4:59:16 PM5/18/22
to Google App Engine

As per the error "backend_connection_closed_before_data_sent_to_client", the backend unexpectedly closed its connection to the load balancer before the response was proxied to the client. Please, follow this link for more information [1].


The GFE generates this HTTP response code[2] when the connection to the backend is dropped during a request.


It is usually caused because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed (10 minute) keepalive (HTTP idle) timeout of the GFE. There are some situations where the backend may close a connection too soon while the GFE is still sending the HTTP request.


The solution for this issue, is to configure the keepalive timeout parameter for your web server appropriately. (In short, set Keepalive to greater than 10 minutes, such as the recommended value of 620.)


For more information, see the reference Google documentation below in link[3][4].


If it keeps failing after you changed the timeout, I’d suggest you contact Support.


[1] https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring#:~:text=backend_connection_closed_before_data_sent_to_client

[2] 502s - "backend_connection_closed_before_data_sent_to_client" responses

[3] https://cloud.google.com/load-balancing/docs/https#timeouts_and_retries    

[4] https://cloud.google.com/load-balancing/docs/https#statusdetail_http_failure_messages 

[5]  https://cloud.google.com/support

Vit Amin

unread,
May 30, 2022, 4:24:33 PM5/30/22
to Google App Engine
"The solution for this issue, is to configure the keepalive timeout parameter for your web server appropriately. (In short, set Keepalive to greater than 10 minutes, such as the recommended value of 620.)"

where should I do it in case of Google App Engine Flexible? how to get this settings?
Reply all
Reply to author
Forward
0 new messages