[ANN] cqrs-server - An opinionated CQRS/ES implementation using Onyx, Datomic, DynamoDB, Kafka and Zookeeper.

[Deon Moolman]

Hi everyone,

I spent some time putting together an implementation of the CQRS pattern in Clojure and wrote an article on it:

http://yuppiechef.github.io/cqrs-server/

It mostly boils down to an Onyx (http://www.github.com/MichaelDrogalis/onyx) configuration, but it's been an interesting journey that I felt is worthwhile sharing.

Feedback really appreciated!

Cheers,

 - Deon

[Aaron France]

Hi,

What are your opinions on Onyx?

What are your opinions on Onyx compared to Storm?

What are your opinions on Onyx deployment?

Aaron

[Alan Dipert]

Thanks for sharing your journey, this was very interesting!  I have written an application in CQRS style using only Datomic and suspect the tools you introduce might have made for a better experience.  I look forward to your future writings.

Alan

[Deon Moolman]

Hi Aaron,

Onyx is still quite young, but incredibly promising. I absolutely enjoy the way that they have teased apart the different bits of distributed systems. I highly recommend getting involved in the project, they're going to do great things.

As for Storm, I haven't really used it so I'm not really qualified to comment. I've stayed away from it mostly because the defspout and defbolt macros made a deep part inside me cringe. Other than that, I'm sure it's a perfectly capable platform and I've heard people doing a lot of great things with it.

Onyx really just translates to a library, at the end of the day. You build your application on top of it and manage firing up your peers inside each process yourself. This gives you as an application developer immense flexibilty. Deployment is outside the scope of Onyx - it assumes you've got that covered. I think that's a very wise assumption, given that the ways to deploy jars are diverse.

Cheers,

 - Deon

[Deon Moolman]

Hi Alan,

Thanks for your feedback - I'm glad you enjoyed it! I will certainly write some more as the project matures :)

Cheers,

 - Deon

[Christopher Small]

I'll chime in with a couple of comments about Storm vs Onyx.

I've used Storm in a production application, so I'm fairly familiar with it. I haven't spent too much time playing with Onyx yet, but will be soon. From what I do know about it and Storm though, I can say the following:

Both Storm and Onyx are similar in that you specify distributed computations via computational topologies. So in general, you can do similar pretty things with them. So for the differences:

• Storm is certainly much more mature at this point.
• At the moment, Storm is much faster, though Michael D. has some plans for stealing some of the performance tricks and intergrating them into Onyx.
• The main difference: As Deon points out, Storm's functionality is heavily built on macros, and rather opaque. In contrast, Onyx embraces using simple data structures to describe the flow of a computation. This makes the specification of computational flow much more modular and composable, to the extant that it's even possible to modify the computational flow at runtime.
• Onyx is built from the ground up in Clojure, for Clojure, whereas Storm has a lot of Java under the hood, and places stronger emphasis on it's Java API than it's Clojure API
• Onyx is moving (has moved? forget now...) to a very clever masterless architecture, while Storm depends on Zookeper, which is a pretty massive piece of software.

If you need something that's battled tested right away, Storm may be your best bet. But I think as it matures (and it's developing quickly and beginning to get production adoption), it's going to win out over Storm, at least within the Clojure community, for the strengths mentioned above. The value of embracing data structures for the sake of composability is well argued and described in Zach Tellman's Always Be Composing talk; this is something that seems to be catching on among Clojurists, and will likely see Onyx gain significant traction.

My two cents...

Chris Small

[Lucas Bradstreet]

Just a small clarification: both Storm and Onyx both depend on Zookeeper. Onyx is masterless as of 0.5.0, however it still requires Zookeeper IN order to write an append only log used by the peers in order to coordinate. In contrast to Storm, Onyx does not have dedicated coordinator nodes (in Storm these are Nimbus nodes). The masterless design is described at http://michaeldrogalis.github.io/jekyll/update/2015/01/20/Onyx-0.5.0:-The-Cluster-as-a-Value.html. 

Lucas

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christopher Small]

Good catch; Thanks for correcting me.

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/hLmaN3sB-Q4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Mike Haney]

Deon,

This is outstanding. I experimented with a similar approach last summer, but struggled with a couple of things. First was what to use for the event store. I considered a few options, like using Datomic for that as well (I.e. annotate transactions and treat the sequence of transactions as the event stream) and even just using Kafka and retaining the logs indefinitely. Both of those options have problems. I like your approach of using a simple KV store - I'm embarrassed that I didn't think of that earlier.

Probably the biggest issue was the overall glue to tie the system together in an extensible way. This was before Onyx was released, and although when I saw Onyx I thought it was promising, I had moved on to a different project and didn't have the time look into it fully. I'm glad to see someone else has done that work.

I think this approach is very promising, and I will be digging into your system at the first opportunity. Thank you so much for sharing this work with the community.

Mike