Liveness and Readiness Probes creating a mess in JBoss + kubernetes 1.6

[Aitor Moragas]

Hello,

Let us assume my JBoss takes a time = 10 time unit to deploy.

Let us assume in my JBoss I have an application running with a HTTP GET /status that returns HTTP 200.

When I try to configure my liveness and my readiness probe, I often fall in a situation of restart loop.

Let me show some examples of initial delay seconds for these two probes:

Test #  -- Initial delay liveness probe -- Initial delay readiness probe -- result

test 1 -- 12 time units -- 8 time units -- OK

test 2 -- 12 time units -- 12 time units -- KO

test 3 -- 20 time units -- 20 time units -- KO

test 4 -- 21 time units -- 20 time units -- OK

test 5 -- 60 time units -- 60 time units -- KO

test 6 -- 60 time units -- no readiness probe -- KO

test 7 -- no liveness probe -- readiness probe -- OK

test 8 -- 12 time units -- 10 time units -- OK

OK : The pod is created and is stable. It is eventually set to ready and traffic can reach the pod.

KO : The pod can't be created, it enters a re-creation loop. Events show liveness failed checks.

Can anyone come with an explanation as to why the liveness probe fails unless the readiness probe has been successful before? The most surprising point is that I actually see JBoss deploying the application yet the liveness probe fails.

2018/06/14 07:22:35.000678 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.4.17.GA (AS 7.5.17.Final-redhat-4) started in 205095ms - Started 197 of 265 services (107 services are lazy, passive or on-demand)                                                
2018/06/14 07:24:08.000377 INFO  [org.jboss.as.server] (Thread-3) JBAS015986: Host Controller shutdown has been requested via an OS signal                                                                                                                                         

Kubernetes version: 1.6.1

[Aitor Moragas]

The issue was with the liveness probe timeout and JBoss deployment.

JBoss creates a lot of resources on the first REST call it receives. And this means the first call has an important overhead that leads to a big response time.

Appropriate timeout value solves the issue.

[Rodrigo Campos]

Great you solved it. And thanks for sharing the solution! :)

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.