ServiceManager managing dependent and independent Services

[Don Madison]

ServiceManager, in com.google.common.util.concurrent, accepts a set of Service instances, and provides methods for starting and stopping the set.

The services in the set should be independent of each other because there is is no assurance that any service will be started before another.

I'd like to be able to say that service A depends on service B, implying that B should be started before A, and similarly, that A will be stopped before B.

I may also have a set of services which are independent of each other, setA, and another set, setB, where all the services of setB should be started before any of setA, and all the services of setA should be stopped before any in setB.

Is there anything like this in Guava?

I propose creating a ServiceDependencyManager, which is similar to ServiceManager but differs in two principle ways:

1) It has two types of constructors, one accepting a Set<Service> and another accepting a List<Service>. The services in the set will be started and stopped in any sequence, possibly concurrently, but the services in the list will be started in the sequence of the list, and stopped in reverse sequence.

2) It implements Service, so a ServiceDependencyManager could be included in a collection of services passed to a containing ServiceDependencyManager.

The principle advantage is avoiding exceptions in the log over which I have no control, caused by services calling other services which have not started yet or have already stopped.

I say I have no control because, even if I tediously ask a service if it is ready before I try to call into it, races allow the answer to irrelevant by the time I attempt to be influenced by the answer.

Thanks,

Don

[Tim Peierls]

Possibly relevant:

If you use Guice you can, to a very limited extent, use Guice dependencies to drive service starts using ConcurrentSingleton scope, as described here:

http://tembrel.blogspot.com/2009/11/concurrently-initialized-singletons-in.html

I should update the post to mention a variety of caveats: No shutdown handling, no way to defer startup, unpleasant interactions between Guice Singleton and ConcurrentSingleton (especially in the PRODUCTION stage).

--tim

On Sat, Dec 1, 2012 at 2:02 PM, Luke Sandberg <lu...@google.com> wrote:

Hi Don,

Internally there have been a few discussions about how to manage inter-service dependencies during startup (and to a lesser extent during shutdown).  This turns out to be a difficult problem from an API perspective and especially from an implementation perspective.

Also, I think there are a few problem with your proposed API. 

1. using a List<Service> to express dependencies would IMHO introduce unnecessary serialization of service startup.  Service dependencies in my experience are not linear but rather a forest of small 1 level trees.  (i.e. there is often a base service and a number of other services that are dependent upon it).  If we wanted to support an explicit dependency management api then i think we would need to model them as a possibly unconnected DAG.

2. Modeling a collection of Services as a Single service turns out to be fraught with issues.  Internally, the original version of ServiceManager used this approach and it made dealing with service failure very difficult which is why ServiceManager very explicitly does not implement Service (e.g. think about how you would implement the state() method in the wrapper service, what if one service spontaneously failed or stopped itself?).

3. Even if you came up with a workable API for managing inter-service dependencies during startup you still have the same issue when dealing with dependencies between service and non-service code which this solution doesn't actually address.  (e.g. consider a Servlet that is accessing a service during its init() method, or during its service() method? how do you ensure that the Servlet isn't initialized by the container until its Service based dependencies have started).

I have a proposed solution to this problem but there isn't consensus that it is a good idea yet (caveat emptor) and it is certainly not a complete solutions, but in my relatively simple use cases it works well and i suspect it would for you as well.

If you have a service with public methods that depend on the service being started to function then implement this method in your service

private void ensureStarted() {

  startAndWait();  // This ensures that we have started

  if (!isRunning()) {

    logger.severe("Service is " + state());

  }

}

and call ensureStarted() at the beginning of every public method that depends on the service being started.  This pattern means that the inter-service dependencies will be discovered lazily at runtime and it also deals well with dependencies between service and non-service code.  There are some issues with it though.

1. It does nothing to cope with correctly ordering service shutdown.  (In my experience, very few Services do anything in their shutdown methods and i have yet to see one that needed to wait for another service to stop before it could stop, so at worst this would be a rare issue).

2. It requires Services to have reasonable behavior once they have FAILED or STOPPED. (or alternatively you can just throw IllegalStateExceptions if it is not RUNNING).

3. If you have a circular dependency between services it will deadlock. (IMHO, this is not so bad, since such a dependency is a programmer error but it is unfortunate that the failure condition is deadlock and not something more reasonable like an exception being thrown).

4. It turns a lot of methods into potentially blocking methods which can be unexpected.

Another option is just to manage the startup yourself.  One of the nice features of ServiceManager is that all its features work correctly even if someone else starts or stops the services.  So if you know your dependencies you could implement this feature without any changes to ServiceManager itself. e.g.

ServiceManager manager= new ServiceManager(ImmutableList.of(baseService, dependentService1, dependentService2));

baseService.startAndWait();

dependentService1.start();

dependentService2.start();

dependentService1.startAndWait();

dependentService2.startAndWait();

// then when you want to stop everything do:

dependentService1.stop();

dependentService2.stop();

dependentService1.stopAndWait();

dependentService2.stopAndWait();

            baseService.stopAndWait();

That code would ensure that things are started and stopped in the correct order. (But it does not handle services which might fail or stop themselves).

Luke

P.S.  I don't have a proof, but i don't actually think it is possible to solve the service dependency issue externally to the services. i.e. if one service has a dependency on another it will probably require a change to the implementation of at least one of the services (and possibly) both to implement both proper startup and shutdown ordering.  So a real solution to this issue may require changes to the service interface / AbstractService. 

--
guava-...@googlegroups.com
Project site: http://guava-libraries.googlecode.com
This group: http://groups.google.com/group/guava-discuss
 
This list is for general discussion.
To report an issue: http://code.google.com/p/guava-libraries/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "guava")

[Don Madison]

Thanks, Luke and Tim for your suggestions.

I definitely agree that my API is unsuitable for a library, compared to engaging a full DAG of service dependencies.

Thanks for the gentle way you expressed "What are you thinking, Don?" :)

I also agree that having a service manager implement the service interface is a bad idea.

I do feel that both service startup and shutdown need to be managed to a large extent outside the services.

As you point out, the application can implement its own service start sequence, and can register a JVM shutdown hook which knows about all services used by the app so it can shut them down in a correct reverse sequence. 

Yes, the existing ServiceManager can start services in an unpredictable sequence, and services can themselves assure that their clients block until the service is ready, but I'd prefer that this tediousness be encapsulated in a new type of service manager, one that manages both startup and shutdown of all services in the JVM.

I work with long running telephone systems, and expect there to be no exceptions in the log unless there's a hardware or software issue which needs investigating. 

There is no apparent strategy for avoiding exceptions in the log when ServiceManager shuts down its services in an unpredictable sequence.

As a remedy, consider an approach which uses Guice to construct all services, ManagedServices, and injects a ServiceProvider for each ManagedService into each class which needs to call methods of the service.

Consider letting ServiceManager2 be a singleton injected into all ManagedServices, and used by the ManagedService during construction to register itself and its dependent services with the manager. This registration allows the service manager to build a DAG of all service dependencies.

The ServiceProvider instances injected into each service client would be wrappers around ServiceManager2 -- the get() method would provide a reference to a ManagedService which has been started, and for which all dependent services have been started.

I've worked in companies which needed the ability to dynamically start and stop running services, and their dependencies, but that's not a feature I'm seeking here.

On the matter of service failures, several approaches may make sense. The simplest, and the one I currently prefer, is to let any service death cause the service manager to shutdown all services and terminate the JVM.

Thanks for your consideration.

Don