Layer 7 SSL load balancer (instances unhealthy)

[Francisco Lopez]

I'm fairly new to GCE and having problems setting up a layer 7 SSL load balancer.

My goal is to have a load balancer front end terminating SSL connections and then calling http on port 8080 of the instances.

I already have setup a regular TCP load balancer forwarding to a couple of instance groups, and the health check works ok (http port 8080)

When I setup the ssl load balancer, all backend services are unhealthy. What I don't understand is why they are since they are using the same instance groups and health check than the regular tcp load balancer.

The only thing I found so far, is that when I describe the backend services, the port shows up as 80 (I'm sure I setup port 8080).

Does anyone have an idea or pointers where to look next ?
Thanks,

Francisco

elessar-2:GCE flopez$ gcloud compute backend-services describe  lms-dev-backend --project labelmatchservice

affinityCookieTtlSec: 0

backends:

- balancingMode: UTILIZATION

  capacityScaler: 1.0

  description: ''

  group: https://www.googleapis.com/compute/v1/projects/labelmatchservice/zones/us-west1-a/instanceGroups/gke-labelmatchservice-dev-default-pool-7294bc92-grp

  maxUtilization: 0.8

- balancingMode: UTILIZATION

  capacityScaler: 1.0

  description: ''

  group: https://www.googleapis.com/compute/v1/projects/labelmatchservice/zones/us-west1-b/instanceGroups/gke-labelmatchservice-dev-default-pool-722a5d86-grp

  maxUtilization: 0.8

connectionDraining:

  drainingTimeoutSec: 0

creationTimestamp: '2016-09-22T16:05:54.613-07:00'

description: ''

enableCDN: false

fingerprint: SjzeAaYY-HI=

healthChecks:

- https://www.googleapis.com/compute/v1/projects/labelmatchservice/global/httpHealthChecks/lmsping

id: '4408135876916927805'

kind: compute#backendService

name: lms-dev-backend

port: 80

portName: lms-port

protocol: HTTP

selfLink: https://www.googleapis.com/compute/v1/projects/labelmatchservice/global/backendServices/lms-dev-backend

sessionAffinity: NONE

timeoutSec: 30

[Kamran (Google Cloud Support)]

Hi Francisco,

When using health check with HTTP(S) Load Balancer, you will need to configure a GCE firewall rule to allow the health check requests that come from addresses in the range 130.211.0.0/22. You do not need firewall rules for Network Load Balancing health checks. Please visit this article for more information.

I hope this helps to resolve the issue. Please let me know if you have further questions about this.

Sincerely,

[Francisco Lopez]

Hi Kamran,

  I had enabled traffic to port 8080, but create another rule anyway. Both instance groups still show as unhealthy.

Where is the place where I can find the logs to indicate the results of the health checks ?

Thanks,

Francisco

[Francisco Lopez]

I have some more information to add, that results in one more question.

When configuring the backend, the description of the instance groups is: "Contains instances that handle requests directed to the backend service through a specific set of ports"

In my case, the instances of the instance groups are kubernetes nodes (I'm using containers) where the pods handling the incoming requests are running. So technically there is nothing listening on port 8080 on the instance group node.

Is Google container engine supported as a backend for Load balancers ?
Thanks

Francisco

[Kamran (Google Cloud Support)]

Hey Francisco,

Since you mentioned that the healthcheck status is healthy when you are using TCP network load balancer on your backend services, but unhealthy when using the same backend services with the HTTP(S) load balancer I assumed that your backend services are listening on the port which you configured for the healthcheck. In general, as described in this article: 

For a health check to be deemed successful, the instance must return a valid HTTP response with code 200 and close the connection normally within the timeoutSec period. If it does so, the health check service returns a status ofHEALTHY for that instance. 

Having said that, since HTTP(S) LB healthcheck requests comes from external IP address range (130.211.0.0/22), your HTTP service on the GKE cluster has to be externally exposed through the Ingress resource and has to listen on node port which you configure for the healthcheck. This article provides example on how to use HTTP Load Balancing with Google Container Engine (GKE).

Regarding to your question about healthcheck logs, unfortunately at the moment logs for the healthcheck resources are not available on Cloud Console. However, there is a post on the GCP blog which provide some tips on how to debug health checks in load balancing. Since having access to the healthcheck logs can help to troubleshoot this kind of issues more efficiently, I opened a feature request for this on GCE public issue tracker. Feel free to star it to get notifications for status updates.

Please let me know if you have further questions about this.

Sincerely,

[Francisco Lopez]

Hi Kamran,

  Thanks for your help.

I could make it work with the ingress resource, however that approach create instance groups not template based which make autoscaling not available. Furthermore, you can't get multizone availability.

Finally, I found a way to make it work at the expense of exposing the container port to the host using:
kubectl run lms-dev --image=gcr.io/XXX/YYY  --port=8080 --hostport=8080 --expose=true

After that, the SSL backend could successfully use the health checks pinging port 8080.

Francisco

[Kamran (Google Cloud Support)]

Hi Francisco,

I'm glad to hear that you could successfully use the health checks with HTTP(S) load balancer and thanks for sharing the solution. It can definitely be helpful to others who have the same question.

Sincerely,