Guava services?

[steve christensen]

I probably should have piped up prior to doing some implementation... 

But, is there any interest in a gwizard module for managing Guava Services?

https://github.com/stevesea/gwizard/tree/master/gwizard-services

and

https://github.com/stevesea/gwizard-example

-Steve

[steve christensen]

Or, to be a little more verbose about it...

The basic idea is that rather than making an application have to know how to startup/shutdown each module, each module can define their own startup/shutdown services that meet their needs.

For instance, in my fork I've changed gwizard-web so that the web server startup is triggered by ServiceManager startup. And, when the ServiceManager is shutdown (say, when the application is exited), the web server is shutdown. 

So, an application that composes many modules just has add  the service module to the injector, then start the provided ServiceManager.class:

Injector injector = Guice.createInjector(

new ExampleModule(),

new ConfigModule(new File(args[0]), ExampleConfig.class),

new LoggingModule(),

new RestModule(),

new HibernateModule(),

                                new ServicesModule());

                // start service manager 

                injector.getInstance(ServiceManager.class)

                        .startAsync()

                        .awaitHealthy();

Any module that added Guava Services to the multibound Set<Service> will have their services started.

Modules can also define Service and Service Manager Listeners with multibindings (see the example at https://github.com/stevesea/gwizard/blob/master/gwizard-services/src/test/java/com/voodoodyne/gwizard/services/example/ServicesModuleExample.java )

-Steve

[Jeff Schnitzer]

It looks great at first glance. I will look closer tomorrow when I'm more awake :)

Thanks,

Jeff

--
You received this message because you are subscribed to the Google Groups "GWizard Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gwizard+u...@googlegroups.com.
To post to this group, send email to gwi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gwizard/9ab2ab7b-feb0-4b09-99f9-ff7d64aeebe3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jeff Schnitzer]

Ok, I've gone through this PR in considerable detail and while I like the direction (and will almost certainly apply it as a starting point), I wonder if Guava Services can be made a little more "guicy" and expose less of Guava's complexity. I'll code up some experiments tomorrow.

I'm thinking something like this:

 * Instead of binding services, provide a Services singleton with an add() method. Any guava service can @Inject it and add(this). Services has a start() method that constructs the ServiceManager and runs startAsync().

 * Provide a simple wrapper of Services, say, Lifecycle with addStartHook(Runnable) and addStopHook(Runnable) methods. Any class can @Inject this, call those methods, and the result will be a new service is generated. Most services won't have any idea that Guava services exists under the covers and the inheritance chain is unmolested.

 * With this, the only special guice behavior an app needs is to bind services as eager singleton. No need for multibinding. This gets nicer with something like Governator's @AutoBindSingleton annotation.

I would love to hear what you think.

Speaking of which, I just stumbled across it for the first time today. @AutoBindSingleton is the only bit that looks appealing to me, but it does look nice. It would be easy to add a similar mechanism as an AutoBindModule. Or anyone could just use Governator as-is.

Jeff

[steve christensen]

I'll have to look at it in more detail after the work day... but, some initial thoughts

• Isolating the knowledge that it's Guava Services/ServiceManager underneath seems like a good idea. 
• I like your idea of a Services singleton that each Service can @inject and do a add(this). The multibinder syntax is awkard and frequently confusing.
• Still trying to wrap my head around your addStartHook/addStopHook suggestion. 

• RE: Governator and autobind. There's also Onami Autobind https://onami.apache.org/autobind/  
• The documentation is awful and scattered. To see real examples of what's now called Onami Autobind, go here: https://github.com/manzke/guice-automatic-injection
• I'm using it on another project along with Guava services.... So, rather than make each module have to deal w/ ugly multibinder syntax, in the implementation class they do:

@Singleton

@Bind(multiple=true, to=@To(value = To.Type.CUSTOM, customs = {Service.class}))

public class MyFancyService extends AbstractIdleService {

• it could also scan for Guice modules. At the moment, a person writing a gwizard  application has to add the maven dependency, and also update their Guice.createInjector() statement. If you used autobind, a consumer of a particular gwizard module could just add the module to their maven dependencies. As long as they configure the Autobind's StartupModule to scan gwizard's package prefix, it'd find whatever modules Gwizard has annotated with @GuiceModule and loaded it into the injector.
• Something else would probably be needed to handle the ConfigModule, since it has constructor params.

• also RE: Governator. Have you seen Netflix's Karyon project? https://github.com/Netflix/karyon    
• It uses Governator and a number of other Netflix OSS projects to build a Dropwizard-esque application
• I'd looked into it a few months ago as a Dropwizard alternative, but at the time the benefits of switching our application away from DW didn't seem worth the effort required

• have you had any thoughts around a 'gwizard-core' or 'gwizard-common' module? That is... something that other gwizard modules can rely on as a central part of a GWizard application. Seems like all GWizard apps have logging, configuration. But, maybe they don't run Jetty at all. 

[Jeff Schnitzer]

I looked a lot closer at the way guava services are being used and they don't seem to be the best approach, at least for the services we have so far. The web stack and jmx stack have their own thread pools; by using guava services, all we're doing is starting and maintaining a thread which starts the other threads. Actually in the case of the jmx stack it's even simpler - JmxReporter.start() just registers a listener. So guava services is monitoring the health of a pair of threads that don't actually do anything.

This kind of behavior seems better modeled with simple addStartHook() and addStopHook() methods, plus a wait() method. Aside from being simpler, everything happens on the main thread and any errors will have a simpler stacktrace. This is not to say that a Guava Services module that rides on top of this basic lifecycle would not be useful, but it wouldn't be the 'backbone' of the system.

I'll work this out in a 'services' branch starting from your PR. I think it will be reasonably attractive. And I think having gwizard-services (gwizard-guava-services?) on top of the backbone might be useful.

I'm a little leery of autodiscovery of guice modules. I suspect more modules will end up with parameters; for example, a hypothetical AutobindModule would take a list of package names as a parameter. Since guice modules don't get added/removed all that often (unlike singletons) so I'm not sure autodiscovery would add a lot of value. Dunno. Would need to experiment.

I'll look closer at the Netflix ecosystem. It looks quite elaborate.

I fully expect some sort of gwizard-common to emerge soon, at the very least as a library jar (less likely as an explicit module). It just hadn't come up yet. This Lifecycle class/interface may be what pushes it over the edge.

I've been pretty happy with the piecemeal approach so far - for example, everything works fine without the logging and configuration modules. It makes the examples clean and easy to understand. I hesitate to make a "core" module or something like that that bundles the other pieces, even if it would reduce the total # of pom dependencies. But if we end up with a huge # of modules, maybe it will make more sense.

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/gwizard/67bf714e-4bd8-4447-97c7-fbb0ca72fe17%40googlegroups.com.

[steve christensen]

For the existing services, it definitely feels dumb. The run thread starts the service then it just waits until shutdown to kill it. But, starting Jetty or the JMXReporter will do I/O and network stuff. By moving the bulk of startup cost to the run() method, the thread that's iterating over each Service serially won't get blocked waiting for a badly-behaving startup to complete.

In my mind the benefit of using Guava Services is its ServiceManager. That's the central piece that handles startup/shutdown/health of all the Services you give it -- whether they're relatively dumb or implement periodic tasks or have many, many threads. On the framework side, you don't have to juggle multiple mechanisms -- you're just fed a Service implementation and hand it off to ServiceManager.

The autodiscovery is weird and worth being leery of. I love it and hate it. I'm using it on another project... Things work. I'm happy with it with regard to DRY principles. But... when a new developer comes into the team and wants to understand how things are glued together, it's a definite cognitive speedbump.

The thing I was wishing I had a 'gwizard-core' for was Configuration. I'd started trying to implement polymorphic jackson deserialization like Dropwizard does for yaml configuration (their service loading Discoverable interface magic). So, I needed a place to create a Guice Provider for a ObjectMapper that'd have all the special stuff added to it.  I was specifically trying to configure different Metric reporters so I'd started tweaking that codebase... Then it bled into Configuration. Then I was tearing my hair out for a couple hours over some user-error I'd made when I hadn't completely understood a code comment. Once I moved on to the next obfuscated Jackson error message, I decided to get a drink and put it aside for a couple days. :) 

[Jeff Schnitzer]

I just ran a few experiments and you're totally right about Jetty; all the network setup is done synchronously in the start() method. That's a compelling reason to parallelize it.

I'm less familiar with mbean servers but from walking through the JmxReporter code it looks like all the heavy lifting is going on somewhere outside of the start() method. Maybe during the generation of JmxReporter? Do I misunderstand how this works?

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/gwizard/d9085918-440a-4aa8-853a-1dd677840442%40googlegroups.com.

[steve christensen]

I didn't profile or analyze the JMXReporter code, I'd just assumed the start() method would be bad. 

Looking at it now... a problem with the code as I submitted it is the build call in the MetricsService ctor:

JmxReporter.forRegistry(metricRegistry).build();

Looks like that build() call is doing doing quite a bit. Getting the platform mbean server, constructing the JmxRporter and JmxListener. 

The start() method is calling addListener(). And it looks like that's iterating over every metric in the system and registering it with the JmxListener.

So, probably the build and the start both belong in the Service's run() thread.

  protected void run() throws Exception { metricRegistry.register("jvm.buffers", new BufferPoolMetricSet(ManagementFactory.getPlatformMBeanServer())); metricRegistry.register("jvm.gc", new GarbageCollectorMetricSet()); metricRegistry.register("jvm.memory", new MemoryUsageGaugeSet()); metricRegistry.register("jvm.threads", new ThreadStatesGaugeSet());

JmxReporter jmxReporter = JmxReporter.forRegistry(metricRegistry).build().start(); doneSignal.await(); jmxReporter.stop(); }

-Steve

[Jeff Schnitzer]

Cool, I'll make that change.

Also - is there a reason why you used AbstractExecutorService instead of AbstractIdleService? AFICT, AbstractIdleService provides the same utility (a new thread for startup and shutdown) without having to muck with latches and keep a thread blocked.

I'll push my work in progress to the branch today.

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/gwizard/1f543c48-cbe1-4f7b-99e3-b22baa786ab1%40googlegroups.com.

[steve christensen]

With AbstractIdleService, you override startup/shutdown. I'd assumed I couldn't do much processing in startUp() that potentially took a long time.

Reviewing the ServiceManager code... I think that assumption was wrong. 

Looks like ServiceManager.startAsync() iterates over each of its Services and calls the Service.startAsync(). Which initiates the service startup, but returns immediately. 

[Jeff Schnitzer]

I was pleased enough with the result of this that I merged it into master. I'm still working on updating the docs, but check out the examples and tell me what you think. Overall it's not too different from your PR. I like guava services quite a bit now that the AbstractIdleService stuff is sorted out.

I made the modules install idempotently so that folks don't have to explicitly install the ServicesModule if they are using Metrics and Web (etc).

One thing is that I put the Runtime shutdown hook in the Run class. I figure if you're using ServiceManager explicitly, bypassing Run, then you can be responsible for registering your own shutdown hook (if you even want one).

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/gwizard/7064801e-e052-4c86-897d-8fb81720d566%40googlegroups.com.

[Jeff Schnitzer]

To get this rolling, I released this to maven as gwizard 0.4.

Cheers,

Jeff

[steve christensen]

Awesome! I really like they changes you've made.