Fabric8-Arquillian - Spin up One Pod Before Another Pod and Check Availability?

[Yong Kim]

We are running integration tests using the Fabric8-Arquillian plugin. The database is in its own pod and the service runs from another pod. 

We were wondering if there's a good way to make sure the database pod spins up (and the database completely starts up within that pod) before the other pod (a REST service) is started. Then, make sure that REST service pod is up and ready to take http requests.

We can issue commands to the database and then issue a check to some health endpoint in the Service pod to see if the services within the pods are up, but that still leaves us needing to hook into the ordering of the pods and most likely do this checking around the same place. Is there a good way to do this with the plugin? 

If not, is there a good starting point in the plugin we could take a look at to add this feature in?

Thanks!

[Ioannis Canellos]

Hi,

there are ways to express the dependencies of you pods.

- you can add a "dependency" property inside your arquillian.xml 

- you can turn the json describing your dependencies into a maven artefact and add it to your project as a test scope dependency.

regardless of how the dependencies are expressed, they will get applied before your current projects kubernetes.json and then the plugin will wait for all

pods and services to become ready (before the tests are run).

In most cases this should be enough. 

[1]: Example of adding dependencies in arquillian.xml

<arquillian xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns="http://jboss.org/schema/arquillian"

  xsi:schemaLocation="http://jboss.org/schema/arquillian http://jboss.org/schema/arquillian/arquillian_1_0.xsd">

  

        <extension qualifier="kubernetes">

               <property name="dependencies">http://somehost/somepath/kubernetes.json</property>

         </extension>

  </arquillian>

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ioannis Canellos

Blog: http://iocanel.blogspot.com

Twitter: iocanel

[James Strachan]

Agreed. Another option in general in kubernetes is to use a liveness probe in your rest service - to check the database is up - which would delay the start of the arquillian test until the rest liveness probe returned true.

Ideally we'd have a way to express service dependencies inside k8s so that k8s knows your rest service can't be live unless the database is live; but there's no way to specify that yet.

--

James
-------
Red Hat

Twitter: @jstrachan
Email: james.s...@gmail.com

Medium: https://medium.com/@jstrachan/
Blog: http://macstrac.blogspot.com/

fabric8: http://fabric8.io/

hawtio: http://hawt.io/

Open Source DevOps and Integration

[Jimmi Dyson]

Use a readiness probe to check whether a pod is ready (started up correctly). Liveness probes are used to check if a pod is still behaving properly. It is a successful readiness probe that adds a pod to the service's endpoints so better to use that.

Cheers
Jimmi

[James Strachan]

Did I say liveness; I meant readiness. Doh!

[James Strachan]

Btw in general with cloud native apps, pods need to des with 

[James Strachan]

Whoops hit send too soon ;).

I was gonna say cloud native apps need to try deal with things starting up in any order & dependencies going down & coming back up later - to make things easier to deploy & manage. Not being ready until you can access your dependencies sounds a reasonable policy; along with retrying requests on services which may fail & get restarted.

I guess it might be nice for pods to terminate with a meaningful error if their dependent services or database is not available for some time; then its easier to grok what's wrong from looking at the logs etc

[Jimmi Dyson]

That is a very good point. Perhaps we should make it easier to test scenarios e.g startup when required services unavailable, services available, services become unavailable, become available again? These are normal things that cloud native stuff needs to handle.

Failing on startup because required services aren't available sounds like an antipattern to me - better to keep trying reconnecting as needed. As Kelsey Hightower puts it: "The application should simply retry the database connection, using some sort of backoff, and log errors along the way. At some point either the database will come online, or your company will be out of business." (good read:
https://medium.com/@kelseyhightower/12-fractured-apps-1080c73d481c).

[James Strachan]

Agreed.

I was wondering if things should just retry all the time; my thought was folks might not spot that the system wasn't actually working at all. I guess thats where logging comes in though; pods should probably just keep retrying. Though given pods get restarted if they fail; its a pretty good hint things are not well. I guess it comes down to logging versus events; is it better to just use logging to indicate things are not well?

[Yong Kim]

Thanks all for the input! It's very helpful and gives me a lot of hope that I can solve these issues. I forgot I had read about the readiness and liveliness pods in the Kubernetes documentation. Do you have a working example integration test where you're using the readinessProbe? 

I have created a simple example and cannot get this to work for my integration tests. I am using the fabric8 vagrant image on my local machine. I've taken a look at SessionPodsAreReady.java and from the looks of it, it uses the pod's "phase" to determine if it's up and running, yet the pod has a "status" with a "condition" and "podcondition" where the "reason" is "ContainersNotReady," presumably because I am using the readinessProbe and it has determined the pod is not ready yet.

If I take it a step further and enable "wait.for.service.connection.enabled" In SessionServicesAreReady.java, it looks for endpoints, but it looks like it's using the internal IP addresses inside the Vagrant machine, which won't be available to me from my dev machine.

Is this how this is all intended to work? If you'd like I can post my example project, but I was wondering if you had a good working example I could look at. I couldn't find any in the ipaas-quickstarts or the fabric8-arquillian plugin - but perhaps I overlooked something.

Thanks!!

[Yong Kim]

I should add that in SessionPodsAreReady, it says the pod's status is "OK" because of the KubernetesHelper method that's used and continues to launch the test instead of waiting for the container to actually be ready.

On Friday, January 8, 2016 at 6:05:53 PM UTC-5, Yong Kim wrote:

[Yong Kim]

Hello,

I've tracked this down and it was not doing proper checking for readiness probes. I've submitted a pull request to take care of this. Before, it was only checking if a pod was running, not if the "ready" condition was "true." Can someone take a look at this?

https://github.com/fabric8io/fabric8/pull/5595

Thanks!

On Friday, January 8, 2016 at 6:05:53 PM UTC-5, Yong Kim wrote: