getting the healthy instances

242 views
Skip to first unread message

Nicolae Marasoiu

unread,
Jun 19, 2015, 6:16:11 AM6/19/15
to consu...@googlegroups.com
Hi,

Currently, my understanding is that there are nodes' serf statuses, service health checks and nodes health checks (those with serviceId="").
In order to filter only the healthy ones, if we enforce that all services have a check at least (at that should include databases, remote services), than calling service health for service name and passing "?passing" should return all service instances that have all the checks green. The next step would be to filter the nodes by health and serf status (not sure of the most efficient way other than query the nodes/checks, filtering to only those with serviceId="" on the client side).

On the other hand, some services currently do not have checks associated: currently redis and many external services, as well as when the publish is done by docker registrator, or mesos services.
So in that scenario, getting ?passing is not ok. In that case currently I get all service instances from catalog, get all health checks, make a group by on the client side, and filter out those having non green statuses. Of course this allows me to do more magic stuff like, if there is no fully green instance, fallback to yellow ones (warning). But the main pain point is that for a simple case - get all fully greens - there is no clear/easy way in, as far as I see. I do not mind making the complex stuff, but discussing with Spring-Cloud-Consul guys, they would opt for a simpler solution as the default instance filtering strategy in their discovery flow.

Please advice,
Nicu

Nicolae Marasoiu

unread,
Jun 19, 2015, 6:32:16 AM6/19/15
to consu...@googlegroups.com
More concretely, I would add a "?passing=false" option, to get the non green checks.
Also, the "?passing" would make even more sense from my perspective on an endpoint that just returns service instances, without checks (which would all be green). It is true however that that would imply a 2-step processing on the agent side, and perhaps it is not the intent.

Armon Dadgar

unread,
Jun 23, 2015, 6:29:07 AM6/23/15
to consu...@googlegroups.com, Nicolae Marasoiu
Hey Nicu,

I’m not sure I understand entirely. If you are using the “?passing” filter, then only the services
with passing health checks or no health checks are returned. That seems like it does exactly
what you are asking for unless I’m missing something.

Best Regards,
Armon Dadgar
--
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nicolae Marasoiu

unread,
Jun 23, 2015, 7:53:00 AM6/23/15
to consu...@googlegroups.com, nicolae....@gmail.com
Hi Armon,

Thanks, that is good.
The only extra needs that I have are:
1. A way to filter in a graceful degradation style i.e. if there are no fully green service instances, to get all but critical ones, together with all checks values, as an example of sharing logic between client and server if possible. For instance instead of passing, having a maxHealth or a minHealth, to enable whitelisting or blacklisting on the client side based on more flexible scheme.
2. Perhaps it is useful to define a node level check e.g. descriptors count, this will be a check on the node and with serviceId="" I understand, and would like to filter all service instances by the node check too. Currently my understanding is that I can do this with separate operations from the client side, is there a way to bundle this in too?

Thank you
Nicu


On Tuesday, June 23, 2015 at 1:29:07 PM UTC+3, Armon Dadgar wrote:
Hey Nicu,

I’m not sure I understand entirely. If you are using the “?passing” filter, then only the services
with passing health checks or no health checks are returned. That seems like it does exactly
what you are asking for unless I’m missing something.

Best Regards,
Armon Dadgar

Armon Dadgar

unread,
Jun 25, 2015, 5:32:48 AM6/25/15
to consu...@googlegroups.com, Nicolae Marasoiu, nicolae....@gmail.com
Hey Nicolae,

There is no way to extend the behavior of Consul currently to support what you are asking for in #1.
The filtering logic is pretty simple and just excludes any critical services. I do think this is something we’d
like to support in the future. Not quite arbitrary logic, but a number of configurable knobs for common
scenarios.

With respect to the second question, this is done by default! The health checks of a service transitively
include the checks on the node hosting the service. So if the HTTP health check for a service is passing,
but the disk utilization check for the node hosting that service is critical, that service will still be filtered out.
So #2 is the default behavior.

In addition to this, you can always consume the HTTP API and do any sort of custom logic you need!
That was one of the motivators behind that API was to allow clients to have rich logic when needed.

Hope that helps!

Best Regards,
Armon Dadgar

From: Nicolae Marasoiu <nicolae....@gmail.com>
Reply: Nicolae Marasoiu <nicolae....@gmail.com>>

Nicolae Marasoiu

unread,
Jun 29, 2015, 4:02:14 AM6/29/15
to consu...@googlegroups.com
Thank you very much:)
So the endpoint is /v1/health/service, I see, right?
What about serf status - is it also automatically factored in here ? (is this filtered by online hosts only)?

Thanks,
Nicu


On Friday, June 19, 2015 at 1:16:11 PM UTC+3, Nicolae Marasoiu wrote:

Armon Dadgar

unread,
Jun 29, 2015, 1:46:40 PM6/29/15
to consu...@googlegroups.com, Nicolae Marasoiu
Hey Nicu,

That is the correct endpoint. The Serf status is automatically populated into a health check and that
is filtered as well (since it is a node level check).

Best Regards,
Armon Dadgar
--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---

You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages