Debugging Load Balancer issues

[Grant Trevor]

I've configured a HTTP(S) Load balancer as per the documentation on https://cloud.google.com/compute/docs/load-balancing/http/ and using Terraform as per https://www.terraform.io/docs/providers/google/r/compute_global_forwarding_rule.html


The health check on my backed service is showing as healthy, and I can see in the server logs that the health check requests are indeed hitting the server and returning a 200.


When I try to access the site via the IP address associated with the Forwarding Rule I'm getting a 502 response with the message:

Error: Server Error

The server encountered a temporary error and could not complete your request.

Please try again in 30 seconds.

I believe this is coming from the load balancer.

The URL map is configured to forward all unmatched (host, paths) to my backend service

In the Load Balancer Logs there are a stack of Warnings for my requests which if I drill down into it looks like it doesn't know which backend service to use, though having checked another load balancer I can this value is empty in the logs as well so may be a red herring.

labels: {

backend_service_name: "" 

target_proxy_name: "webhost-https-proxy" 

zone: "global" 

project_id: "my-project" 

url_map_name: "webhost-https-url-map" 

forwarding_rule_name: "webhost-https-forwarding" 

}

Anyone have any insight into what might be going on, what more I should be looking at?

[Kamran (Google Cloud Support)]

Hello Grant,

Referring to RFC7231: The 502 (Bad Gateway) status code indicates that the server, while acting as a gateway or proxy, received an invalid response from an inbound server it accessed while attempting to fulfill the request.  Therefore, it's not necessarily a gateway (here the Load balancer) that can cause the issue but this error can be generated when there was difficulty reaching an internal service.

Having said that, are you seeing this error when deploying your HTTP(S) load balancer using Terraform? Or setting up a HTTP(S) LB manually to target the same backend services will also cause the issue?

My colleagues answered similar questions on these stackoverflow links [1] & [2]  that I'd like to invite you to take a look to see if the answers are helpful for you. After reviewing those answers, if you were still not able to resolve the issue, feel free to send me your project ID and LB name (via private message) and I'll be glad to assist you on resolving this.

Sincerely,

[Grant Trevor]

Thanks Kamran

I've reviewed those items and am still having issues.

The load balancer logs are showing the error as: failed_to_connect_to_backend

The healthchecks are passing.

If I access the GCE instance directly via it's External IP the site that it hosts loads as expected.

I'm wondering if it's the URL Map Configuration it would seem it's unable to route the requests to the configured backend service.

[Kamran (Google Cloud Support)]

The "failed_to_connect_to_backend" error message indicates that load balancer is failing to connect to backends.

Sure, investigating URL map rules is good point to start. In fact, when a request comes into the load balancer, it is routed to backend services based on configurations in a URL map. Misconfiguration of URL Map can cause requests go nowhere and never reach to backend. Please note that healthcheck does not use URL Map rules to check backends and therefore a healthy status does not mean load balancer is properly configured but shows that backends are running and healthy.

I'd suggest reviewing your LB's URL map to make sure that Host rules, Path matchers and Path rules are properly defined and comply with descriptions in this article. Also, try increasing Timeout value of LB to a higher value to see if this value is not the source of issue. The default value is 30s.

[Grant Trevor]

In the end it was mis-configuration of the Instance Group and Backend Service.

I hadn't added a Named Port to the Instance Group for https all working now.

Cheers

On Sunday, February 12, 2017 at 9:36:17 AM UTC+10, Kamran (Google Cloud Support) wrote:

The "failed_to_connect_to_backend" error message is self-explanatory that indicates the load balancer failed to connect to the backend services.

Sure, investigating URL map rules is good point to start. In fact, when a request comes into the load balancer, it is routed to backend services based on configurations in a URL map. Misconfiguration of URL Map can cause requests go nowhere and never reach to backend service.  Please note that healthcheck does  not use URL Map rules to check backend services and therefore a healthy status does not mean load balancer is properly configured but shows that backend services are running and healthy.

[Kamran (Google Cloud Support)]

Hi Grant,

Glad to hear that you resolved the issue and thanks for sharing your findings. Definitely this can be helpful for other users who experience the same issue.

Cheers,

[Joseph Wang]

Hi Grant

It seems I met same issue here when I use both terraform and gcp to create load balancer for our service. Based on your statement, is that mean you have to created a named_port in your group manager recourse. And when you create backend service you set that backend service port_name to that named_port's name in group manager?

Best,

Joseph

[Damon Palamarek]

Hello Joseph,

For “internal HTTP(S), external HTTP(S), SSL Proxy, and TCP Proxy load balancers, backend services must have an associated named port if their backends are instance groups.” So if one is not configured on the backend for your instance group then yes, you will need to add a named port to the backend/instance group.

Cheers,

[Joseph Wang]

Thanks! This solved my issue alot