Some initial questions

[Bruce Mitchener]

Hello!

Taking a look at RockSteady after seeing it on the blog / Twitter and ran into some questions...

Performance. Do you anything that you can share about the performance of the system?  Obviously it will depend on a lot of factors (how many rules are running in Esper, how complex they are, what type of hardware, etc), but some rough ideas would be interesting to know.  Are we talking about a thousand metrics per second? One hundred? Ten thousand?

Integration. I'm currently using Cloudera Flume to aggregation and collection of data. I was looking at integrating a CEP system.  If I (or someone) were to want to integrate the core of RockSteady with a different routing system (not RabbitMQ), would such a set of patches be welcome?

Licensing. You're integrating Esper which is GPLv2 with Apache 2.0 licensed code. I thought that there were issues there? Are you using the commercially available version of Esper in your own deployments?

Contributing. If I have some changes to submit, do you require any sort of paperwork or should I just open an issue and attach the patch?

I was thinking of submitting a couple of patches:

 * Some imports can be cleaned up. (There are imports of CEP stuff in classes that don't need it.)

 * The usage of slf4j could use the string interpolation that it supports so that debug logs aren't as expensive: logger.debug("..." + foo.toString()) can be logger.debug(".... {}", foo)

Thanks!

 - Bruce

[Mark Lin]

Hi Bruce, replies inline. Great questions, btw.

> Performance. Do you anything that you can share about the performance of the
> system?  Obviously it will depend on a lot of factors (how many rules are
> running in Esper, how complex they are, what type of hardware, etc), but
> some rough ideas would be interesting to know.  Are we talking about a
> thousand metrics per second? One hundred? Ten thousand?
> Integration.

Esper doc has a chapter on it.
http://esper.codehaus.org/esper-2.1.0/doc/reference/en/html/performance.html

It comes with a performance kit, result from codehaus is posted here
http://docs.codehaus.org/display/ESPER/Esper+performance

For us, we run about 85K metric through rocksteady with about 20
statements residing on two core instance in our private cloud and it
uses 15-20% cpu. But as you said, it has a lot to do with complexity
of the statements.

To scale beyond single Esper instance, we can separate statements into
layers and with system like rabbitMQ(or any queue system), it should
be easy to do message passing between Esper instances.

I'm currently using Cloudera Flume to aggregation and
> collection of data. I was looking at integrating a CEP system.  If I (or
> someone) were to want to integrate the core of RockSteady with a different
> routing system (not RabbitMQ), would such a set of patches be welcome?

Yes, such patch will be welcomed. Others with their own metric/event
could benefit from these patches.

Adding your own routing system should just need a service in "router",
src/main/java/com/admob/rocksteady/router/MessageManager should get
you started quickly.

Let me know if I can help.

> Licensing. You're integrating Esper which is GPLv2 with Apache 2.0 licensed
> code. I thought that there were issues there? Are you using the commercially
> available version of Esper in your own deployments?
> Contributing.

Yes, GPL v2 and Apache v2 aren't compatible. So I took Esper out of
distribution binary. For source build, maven will fetch it. For
running distribution, start script will check existence of Esper and
try download it if necessary.

> If I have some changes to submit, do you require any sort of
> paperwork or should I just open an issue and attach the patch?

Google requires a CLA(Contributor License Agreement) form which makes
sure no one gets hurt when submitting patches. :) The form does not
transfer copyright: the contributor retains his/her ownership in the
work. As an individual, you can submit it electronically here:
http://code.google.com/legal/individual-cla-v1.0.html

> I was thinking of submitting a couple of patches:
>  * Some imports can be cleaned up. (There are imports of CEP stuff in
> classes that don't need it.)
>  * The usage of slf4j could use the string interpolation that it supports so
> that debug logs aren't as expensive: logger.debug("..." + foo.toString())
> can be logger.debug(".... {}", foo)

yes and yes, both changes will be good to have.

> Thanks!
>  - Bruce
>

Thank you!
Mark