How grpc-health-probe (gRPC health checking on Kubernetes) differentiates between liveness and readiness probes

[abhay paroha]

Hi,

I am writing one grpc service and using gRPC health checking on Kubernetes (https://github.com/grpc-ecosystem/grpc-health-probe). In my server, I added different implementation (one for liveness and other for readiness) of endpoints. I am wondering how this probe utility binary differentiates between liveness check vs readiness check? There should be some other way to define it in yaml, not just ["bin/grpc_health_probe", "-addr=:8801"]

server = ServerBuilder.forPort(port)
  .addService(new GrpcModuleHealthCheck())
  .addService(new GrpcModuleReadinessCheck())
  .addService(ProtoReflectionService.newInstance())
  .build.start

In kubernetes deployment yaml, I am using below configurations

    livenessProbe:
      failureThreshold: 3
      exec:
        command: ["bin/grpc_health_probe", "-addr=:8801"]
      initialDelaySeconds: 240
      periodSeconds: 20
      successThreshold: 1
      timeoutSeconds: 15
    readinessProbe:
      failureThreshold: 3
      exec:
        command: ["bin/grpc_health_probe", "-addr=:8801"]
      initialDelaySeconds: 20
      periodSeconds: 20
      successThreshold: 1
      timeoutSeconds: 15

I just tested and found "GrpcModuleReadinessCheck" (the health class which I added last) implementation is taking effect when I just exec my kubernetes pod

kubectl exec -it <MY_POD_NAME> -- /bin/bash

bash-4.4$ ./grpc_health_probe -addr=localhost:8801
status: SERVING

I want to execute different logics for liveness and readiness probes but right now the health implementation added at last in grpc server is taking effect. How to achieve it?

Thanks & regards,

Abhay Dutt Paroha

[apa...@slb.com]

Anyone has idea? How to implement it?

Thanks,

Abhay 

[Eric Anderson]

The health probe doesn't distinguish between the two. It is a binary that is run, and if you use the same arguments you will get the same results. The binary only accesses the Health service (/grpc.health.v1.Health/Check as mentioned in its README).

One option is you can pass the `-service` command line argument to change the "service name" requested by the Check service, for one of the two checks. The default service is empty string. It's unclear what implementation of the health service you are using, but if using the premade-implementation, you can call the HealthStatusManager.setStatus(serviceName, status) as appropriate.

[apa...@slb.com]

Hi Eric,

Thanks for the reply.

I am writing grpc server in scala. I have 2 classes (one for liveness and other for readiness) and both are implementing HealthGrpc.HealthImplBase of grpc-java.

In both classes, I implemented override def check(request: HealthCheckRequest, responseObserver: StreamObserver[HealthCheckResponse]) method then in my server code, I added both class, obviously the last one is taking effect as there is no way to distinguish between both of them. I want to run both liveness check and readiness checks (both has different logic) for single service. How -service argument will help here?

server = ServerBuilder.forPort(port).addService(wssProto.WorkflowEngineServiceGrpc
  .bindService(<MY_SERVICE>) # I want to run liveness and readiness checks for this grpc service.
  .addService(new GrpcServiceHealthCheck())
  .addService(new GrpcServiceReadinessCheck())
  .addService(ProtoReflectionService.newInstance())
  .build.start

[Kun Zhang]

There can only be at most one service registered to the server at the same time. If more than one services with the same name are added to addService(), only the last one will take effect.

If you want to differentiate two different kinds of checks, you have to make them into a single HealthGrpc.HealthImplBase implementation, then you can send different service names as Eric mentioned above, and in the HealthGrpc.HealthImplBase implementation use different service names to signify the type of check requested.