Service Oriented Architecture anybody?

[Paul Robinson]

Hi all,

I'm asking NWRUG this despite being in London these days, because I know some of you might have dealt with similar setups. Oh, and I've been to far, far more NWRUG meets than I have LRUG. :-)

I'm currently trying to get my head around an app that has an architecture heavily inspired by this book:

http://www.amazon.co.uk/Service-Oriented-Design-Rails-Addison-Wesley-Professional/dp/0321659368

It is implemented using https://github.com/typhoeus/typhoeus in various clients talking to several different JSON API endpoints provided by multiple Rails apps, and right now I'm feeling quite debilitating pain points around testing, adding new features quickly, deployment in the sense we have a lot of code to keep up in the air, etc. These are all pain points I feel would be avoided with a more "monolithic" app.

I can fix them without changing the architecture, but it feels like there isn't a bigger picture issue going on here.

I am torn. Clearly, SOA is the way to go if scaling is a major issue and we want to swap in/out components. However I am also a fan of convention over configuration and some of the hoops you jump through with Typhoeus feel incredibly unconventional and very un-Rails like. Un-Ruby like in some cases. Implementing a 20-30 line method that queues up API calls and then runs them in a threaded asynchronous fashion is very clever, it's just not the way I've trained myself to write code these days...

I have a choice:

1. Collapse things back down into a more monolithic app. Almost all components are Rails with the need to talk to some third-party client applications via APIs which we could implement with some nested controllers off to the side of a conventional monolithic application

2. Realise it's unfamiliarity on my part and push through to a greater understanding of why this setup is so awesome and why what I think are code smells are in fact beautiful fragrant bouquets.

3. Perhaps push through on re-factoring what we have so that it's the same architecture but refactored code, smaller methods, etc., or perhaps start looking for other implementations that are "more Rails" like than what we have right now.

I was wondering what your collective experiences were with SOA and if there was any best practice reading you might suggest, or indeed any comments/insights you've had on trying to get bigger SOA Rails apps dancing nicely.

Thanks,

Paul

[Francis Fish]

Hi Paul

Sounds like a bad case of premature optimisation to me. There are some gems knocking around where you can record and replay responses from external API calls (not used them in an age so ... guys?) that may help you write some tests.

You could also look at folding it all back in but using the Rails 3 middleware stack. That might be less up front rewriting - but without seeing the app it's hard to say if it would be worth the hassle.

I was listening to the Ruby Rogues Sandi Metz episode and she was saying that you can do things like identify the piece you need to change, turn off the old code, and do a functional replacement. As in, don't rewrite, but write it for the functionality revealed by the working app, using it as a spec.

I did a lot of SOA work when it was new 10 or so years ago and it always seemed more trouble than it was worth, but we are talking Java/Oracle land where doing anything was pretty hard. When you need to scale, then you need to consider SOA. I wonder if it was cast that way for resilience? As in, you can gracefully replace failed pieces with sensible defaults and the site stays up, plus allow graceful degradation of service?

There's also the 5 9's problem - the whole system's reliability becomes the product of the probability of failure for each component instead of just the most vulnerable piece and without that resilience architecture in place it hurts even more.

Not sure if my rambling's been any help, but I'd rewrite or recast it all. You don't know where the pain is until you feel the pain. Before that it's mere speculation.

Thanks and regards,

Francis

Follow me on twitter https://twitter.com/fjfish
Blog at http://www.francisfish.com
(you can also buy my books from there!)
CV http://www.pharmarketeer.com/francis.html

Comfort the frightened, coach the clueless and teach the uninformed. Seth Godin

--
You received this message because you are subscribed to the Google Groups "NWRUG" group.
To post to this group, send email to nwrug-...@googlegroups.com.
To unsubscribe from this group, send email to nwrug-member...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nwrug-members?hl=en.

[doug livesey]

I don't think that 'un-rails-like' is necessarily a bad thing, to be honest. Your pain points there sound rather like getting past the point of Rails conventions and needing a new paradigm, which you've recognised.

One thing I like about the hexagonal Rails approach that Matt Wynne has been going on about is that its a way of thinking about a much more modularised way of developing (and testing) rails apps. It's not SOA, but SOA leads pretty easily and naturally from it. And it really nicely controls your dependencies (and makes you think about and control your dependencies, rather than let Rails include everything, which is *huge*!).

There's probably a more elegant way of doing it, but currently I launch stub Rack apps to mock any services I need in my integration tests. And I create service objects/libraries to call those services, so that they can easily be stubbed and mocked for unit testing.

The only thing missing there, potentially, is a meta-integration test, where maybe you want to test the whole, distributed stack, with real or staging instances? I've not got to the point of needing to do that, yet, but I imagine that cucumber could do that pretty easily.

But before you distribute everything to its own app, the Hexagonal Rails approach would give you cleanly defined modules that could then much more easily be evolved out into separate services as the need arose.

I've no idea if that's any help, either, but I can hope. :)

Cheers,

   Doug.

[Paul Robinson]

On 21 Jan 2013, at 17:41, Francis Fish <franci...@gmail.com> wrote:

> Sounds like a bad case of premature optimisation to me. There are some gems knocking around where you can record and replay responses from external API calls (not used them in an age so ... guys?) that may help you write some tests.

Yup, vcr is helping us out in a few key areas. We've been doing a lot of work with this to help with CI which is overall in good shape but lacking coverage in a few areas because devs hadn't got vcr plugged in previously and therefore putting any tests in place that Jenkins would want to run would cause us all sorts of issues.

> You could also look at folding it all back in but using the Rails 3 middleware stack. That might be less up front rewriting - but without seeing the app it's hard to say if it would be worth the hassle.

I think that's definitely an option, and I believe some parts of the app have already been done that way - there's some legacy here originally developed in 2.x that has been pushed up to 3.x by the looks of it, but I think time ran out on pushing through the re-factoring to the fullest extent.

As to whether it would be worth the hassle, it's all about finding balance I think. If the end result is "we can ship new features quicker" or "the platform will be more stable", it's easy to justify.

If it's "I sense we could use this funky stuff here because it's cool" or "I feel like this is technical debt but I'm not sure", it probably isn't. :-)

> I was listening to the Ruby Rogues Sandi Metz episode and she was saying that you can do things like identify the piece you need to change, turn off the old code, and do a functional replacement. As in, don't rewrite, but write it for the functionality revealed by the working app, using it as a spec.

Yes, that's an approach I was considering, and coupled with comprehensive Cucumber tests, I'm thinking we might have quite high confidence in going for that style of refactoring.

In essence if we have not just happy paths but unhappy paths all covered and we keep an eye on carefully moving views and assets into the new structure mostly untouched at this stage, we could be quite brutal with what is going on in terms of refactoring behind the scenes.

> I did a lot of SOA work when it was new 10 or so years ago and it always seemed more trouble than it was worth, but we are talking Java/Oracle land where doing anything was pretty hard. When you need to scale, then you need to consider SOA. I wonder if it was cast that way for resilience? As in, you can gracefully replace failed pieces with sensible defaults and the site stays up, plus allow graceful degradation of service?

Problem is, I'm not 100% certain we have that resilience benefit here, as deploying more servers within a given tier/function is currently a little problematic, but that's a separate issue relating to underlying metal than it is SOA.

We're working on fixing that now as a separate activity, but my gut instinct is telling me this is a premature optimisation.

> There's also the 5 9's problem - the whole system's reliability becomes the product of the probability of failure for each component instead of just the most vulnerable piece and without that resilience architecture in place it hurts even more.

Indeed. I think we have some quite brittle couplings in here - there is one key component that shouldn't be key to everything but is elevated to a lofty status of determining if the entire app runs or not through our current setup, so if it goes down the entire app basically grinds, opposed to my preference of a bunch of controllers simply shouting out exceptions in a little-visited corner leaving the rest of the architecture up.

> Not sure if my rambling's been any help, but I'd rewrite or recast it all. You don't know where the pain is until you feel the pain. Before that it's mere speculation.

On the development front, there is pain right now. I thought about whether this pain is my unfamiliarity with the code base, but I am increasingly convinced that SOA is hard to develop tests for quickly, which means it is hard to add features for quickly, it's hard to refactor quickly and it's hard to bug fix quickly.

When it's hard to be agile, when it's hard to push BDD, when it's hard to fix issues, there's probably something wrong somewhere. I'm currently debating whether the difficulty I'm stood in front if simply a small bump or a wall, and I'll let you know what we end up doing I'm sure, preferably over a pint in a pub that serves beer with a decent head as opposed to the flat muck we get down here...

Paul

[Paul Robinson]

On 22 Jan 2013, at 11:24, doug livesey <bio...@gmail.com> wrote:

> I don't think that 'un-rails-like' is necessarily a bad thing, to be honest.

It depends on your "religion". If you think that the power of rapid Rails development is embodied by the mantra "convention over configuration", then you are losing a lot of that power when you stop following the conventions.

If you spend a lot of time configuring things to get a Rails app working, you have to realise you're using Rails incorrectly.

> Your pain points there sound rather like getting past the point of Rails conventions and needing a new paradigm, which you've recognised.

Maybe. I'm not sure that we need a new paradigm. An MVC framework written in Ruby with a persistent SQL storage back-end with Models taking on some sort of ORM responsibility is what Rails is brilliant at, and that's what I think we need here.

It's possible we could look at other options - I scratch my chin thinking about node.js a lot at the moment as we're heavily event-driven as an architecture - but I am not convinced we need to involve new paradigms here out of necessity.

> One thing I like about the hexagonal Rails approach that Matt Wynne has been going on about is that its a way of thinking about a much more modularised way of developing (and testing) rails apps. It's not SOA, but SOA leads pretty easily and naturally from it.

Hexagonal is something I am intrigued with, the pattern looks like it might help us a little in some areas, but I have not tried implementing anything with the pattern so not sure if it'll suit us. Anybody here got direct experience of it and the pain points/successes it brings to the party?

> And it really nicely controls your dependencies (and makes you think about and control your dependencies, rather than let Rails include everything, which is *huge*!).

Got any good write-ups anywhere on Rails implementations of hexagonal that demonstrates that?

> The only thing missing there, potentially, is a meta-integration test, where maybe you want to test the whole, distributed stack, with real or staging instances? I've not got to the point of needing to do that, yet, but I imagine that cucumber could do that pretty easily.

I'd argue that testing the whole stack is the most important test of all. :-)

> But before you distribute everything to its own app, the Hexagonal Rails approach would give you cleanly defined modules that could then much more easily be evolved out into separate services as the need arose.

I'll dive into that approach in more detail and maybe try refactoring a few components and see what it buys us.

Thanks,

Paul

[Will Jessop]

On 22 January 2013 12:22, Paul Robinson <pa...@32moves.com> wrote:
> On 22 Jan 2013, at 11:24, doug livesey <bio...@gmail.com> wrote:
>> I don't think that 'un-rails-like' is necessarily a bad thing, to be honest.
>
> It depends on your "religion". If you think that the power of rapid Rails development is embodied by the mantra "convention over configuration", then you are losing a lot of that power when you stop following the conventions.
>
> If you spend a lot of time configuring things to get a Rails app working, you have to realise you're using Rails incorrectly.

IMO this is valid for setting up a rails app, not so much for writing
an app in it. Sure, RESTy MVC is going to work well with some of your
app, but most apps beyond trivial ones are going to stray from the
paradigm in some way. I think the key is to work out where to do so.
SOA'ing everything is probably as wrong as keeping everything on the
Rails track (ha!).

>> Your pain points there sound rather like getting past the point of Rails conventions and needing a new paradigm, which you've recognised.
>
> Maybe. I'm not sure that we need a new paradigm. An MVC framework written in Ruby with a persistent SQL storage back-end with Models taking on some sort of ORM responsibility is what Rails is brilliant at, and that's what I think we need here.

That's what I use in the app I'm writing right now, except for the bit
(a controller) that communicates over a TCP socket to a job server to
run synchronous tasks on other machines. That's not Railsy at all, but
most of the app is.

> It's possible we could look at other options - I scratch my chin thinking about node.js a lot at the moment as we're heavily event-driven as an architecture - but I am not convinced we need to involve new paradigms here out of necessity.

If you're already using Ruby (and you *really* need an evented server,
most people don't) you should consider EventMachine, unless you really
need JS there's no reason I can see to choose Node. You can always
integrate an evented system into your existing setup, no need to rip
everything out and start again.

Will.

--
Will Jessop
System Administrator
37signals

[Colin Densem]

Hi Paul,

There is nothing wrong with SOA, or to be inspired by the ideas, which are not that radical to programs calling programs pre web. The book you're reading, I read the early beta, some aspects I didn't concur, but thats the joy of this work.

Where I've used it the real pain has always been around testing. Versioning of the endpoints is I'd say mandatory, coupled with the most fragile of assumptions, human nouse. Actually deploying it, designing it I find a little easier when separated to logical points.

The pain on testing, naively take two apps, modify the service endpoints, rewrite tests and both would pass, but neither will work together. Tools out there such as vcr, et al will only work if the api contracts don't change. Writing / modifying contract tests then should trigger a thought in the developers mind,  why am I changing this and what's the impact… you could point your CI to talk to a real staging set of services...

Overall application/implementation complexity is increased, but individual applications are much easier to understand and of course can be taken out and replaced with <insert framework/language>. That alone could be a powerful asset.  Snowflake applications benefit greatly as it's actually reducing code repeated between them. As Francis says it depends on the app/vision.

---

Where I introduced the concepts, it had an impact, yes it cost dev time, yes it was questioned by many, including myself. The site we inherited consisted of two 'monolithic' applications a front (perl) and rear(rails). A new front end; in rails,  was coming, along with a different data structure to the rear, different lookup ids, sequencers etc… coupled to that, touch one the other would invariably break in some unforeseen way.  1000+ line models, scant tests, pain written all over it and our faces.

We introduced services and middleware, message queues to handle interfaces/contact points between the applications, e.g. passing an order into the backend. Another was a feature with several gig of effectively transient data in the core application, it impacted everything, db IO, backups, restore, performance etc… that was obvious to pull out and replace with service calls. In the extraction we could also identify the features; nobody truly knew, write tests and if that service/db failed, it didn't take the site down. It was far leaner so could actually be improved, about 99.9% improvement by time we'd finished on it. As an aside it also meant we could reduce our PCI exposure, so PCI only covered core apps, push the non core off to different servers/datacentres if need be.

To summarise my thoughts on SOA…

• Monolithic or single app is the rails way, SOA totally depends totally on your app. I suspect most apps written are actually fine and SOA is overkill. 
  
• Take a small non-core aspect and build out from there.
  
• You app will be more stable all in one. 
  
• You can drive performance and agility faster on separated applications, more flexibility.
  
• Get it wrong and it's your sea anchor, same goes for going to far.
  
• Aim for <10ms response times, anything over 50ms investigate and tune (this is where cost comes in)
  
• If behind nginx, memcache can be your best friend… http://wiki.nginx.org/HttpMemcachedModule use the api uri as the memcache key ;) Caveat: all data must be same type.
  
• Refactoring has to be completed, a mistake I made was not getting enough to time to clean up, this left confusion and half open doors. 
  
• Do it or do not, but be aware it's going to take longer and that may leave the product owner scratching head...
  

Apologies for length, off to read up on hexagonal, missed that somehow.

Hope this helps in some way.

Colin

[doug livesey]

> Anybody here got direct experience of it and the pain points/successes it brings to the party?

It's all pretty new stuff, so my main liking is that it makes sense to me and gives me another boost in the complexity I can deal with -- I need all the help I can get! :)

But I'm currently migrating a very old Rails app to a very new one, and the bits that are broken apart like a hexagonal app are a joy to transfer over -- drag & drop the files and specs! But all the bits that use Rails conventions feel like pulling teeth -- everything is mixed up with everything else, and so everything is impacted by every small and large change to Rails conventions.

(With regards to everything being mixed up with everything else, one of the biggest culprits there is ActiveRecord! I'm finding going back to having different modules handle fetching, updating, and presenting information (even if they're only limiting wrappers) is much easier to maintain cleanly.)

> Got any good write-ups anywhere on Rails implementations of hexagonal that demonstrates that?

Not really, sorry, but if you have namespaced classes and modules under lib/ (for example) that handle as much of your business logic as possible, and you explicitly require them whenever you need them, then you're already taking responsibility for them. This means you can test them *really* quickly, and also that you can tell at a glance what is in play in any given bit of code, rather than just a sort of fuzzy, rails-y 'everything'.

I find myself, generally and on more complex projects, being bitten by Rails magic more often than helped out by it. But that could just be me. It often is. :)

[Francis Fish]

On Tue, Jan 22, 2013 at 1:24 PM, Colin Densem <colin...@summit360.co.uk> wrote:

Where I've used it the real pain has always been around testing. Versioning of the endpoints is I'd say mandatory, coupled with the most fragile of assumptions, human nouse. Actually deploying it, designing it I find a little easier when separated to logical points.

+1 for versioning, if I remember right we had to make sure the version was in the URL 'cos nginx doesn't include params in the cache key. In fact the whole service call, somehow? Memory fades, it's been 18 months.

Apologies for length, off to read up on hexagonal, missed that somehow.

http://rubyrogues.com/078-rr-hexagonal-rails-with-matt-wynne-and-kevin-rutherford/

+1 

[Francis Fish]

Ah fergit to mention that versioning really helps if you publish the API's to 3rd parties too.

Thanks and regards,

Francis

Follow me on twitter https://twitter.com/fjfish
Blog at http://www.francisfish.com
(you can also buy my books from there!)
CV http://www.pharmarketeer.com/francis.html

Comfort the frightened, coach the clueless and teach the uninformed. Seth Godin

[Graham Ashton]

I've got a write up of how I refactored a Rails controller with some of the hexagonal ideas here:

https://www.theagileplanner.com/blog/building-agile-planner/refactoring-with-hexagonal-rails

In that post I talk about the observer pattern (not to be confused with the "observer" code in Rails).

I learnt to love observers on the biggest project I've ever been involved with (Python, with no web stuff in sight), which was very SOA, with 7 different services sending messages to each other over a home grown message bus. It was big, but the responsibilities of each component were clear.

Paul - I haven't chipped in on this thread until now because I wasn't sure how useful I could be without a bit more context. Would it be easy to flesh out what the system does, and what the various services involved are responsible for?

Francis Fish <franci...@gmail.com> wrote:

On Tue, Jan 22, 2013 at 1:24 PM, Colin Densem <colin...@summit360.co.uk> wrote:

Where I've used it the real pain has always been around testing. Versioning of the endpoints is I'd say mandatory, coupled with the most fragile of assumptions, human nouse. Actually deploying it, designing it I find a little easier when separated to logical points.

+1 for versioning, if I remember right we had to make sure the version was in the URL 'cos nginx doesn't include params in the cache key. In fact the whole service call, somehow? Memory fades, it's been 18 months.

Apologies for length, off to read up on hexagonal, missed that somehow.

http://rubyrogues.com/078-rr-hexagonal-rails-with-matt-wynne-and-kevin-rutherford/

+1 

--

[Paul Robinson]

On 22 Jan 2013, at 18:35, Graham Ashton <gra...@effectif.com> wrote:

> I've got a write up of how I refactored a Rails controller with some of the hexagonal ideas here:
>
> https://www.theagileplanner.com/blog/building-agile-planner/refactoring-with-hexagonal-rails

I read that this afternoon, planning to go through it again with my own examples tomorrow.

> I learnt to love observers on the biggest project I've ever been involved with (Python, with no web stuff in sight), which was very SOA, with 7 different services sending messages to each other over a home grown message bus. It was big, but the responsibilities of each component were clear.

How synchronous did this message passing need to be? That sounds like for our application a deep dive into latency hell if not managed carefully. I'm already fighting that on one part of the setup with some controller actions on the website needing tens of seconds to do their thing.

> Paul - I haven't chipped in on this thread until now because I wasn't sure how useful I could be without a bit more context. Would it be easy to flesh out what the system does, and what the various services involved are responsible for?

It might be difficult to do that on a public list and to maintain commercial confidences. Are you likely to be at an LRUG or NWRUG any time soon?

Failing that, I might be able to be a bit more explicit off-list.

Thanks,

Paul

[Graham Ashton]

On 22 Jan 2013, at 20:11, Paul Robinson <pa...@32moves.com> wrote:

>> I learnt to love observers on the biggest project I've ever been involved with (Python, with no web stuff in sight), which was very SOA, with 7 different services sending messages to each other over a home grown message bus.
>

> How synchronous did this message passing need to be? That sounds like for our application a deep dive into latency hell if not managed carefully.

It varied. Some services didn't really need to handle their work and spit messages onto the message bus particularly quickly, while others needed to act fast as the consequences of their actions were reflected immediately in the UI of a desktop app (which was itself one of the 7 services).

Where latency was an issue we had the option to take an RPC approach (essentially the observer pattern again, but over the network), but because resilience and the need to keep processing incoming data was important (irrespective of whether the GUI was able to keep up), we used it very sparingly.

> I'm already fighting that on one part of the setup with some controller actions on the website needing tens of seconds to do their thing.

That sounds like it needs re-doing; is the task suitable for a background job?

>> Would it be easy to flesh out what the system does, and what the various services involved are responsible for?
>
> It might be difficult to do that on a public list and to maintain commercial confidences. Are you likely to be at an LRUG or NWRUG any time soon?

I'll be at the next NWRUG (21st Feb). Not planning on being in London again for at least a few weeks, but I am due a trip in a month or so. It's unlikely to coincide with LRUG though.

If you're up in Manchester feel free to drop into the office for half an hour. An off list email is also fine, but it's not the most likely route to me making relevant suggestions!

Graham

--
Graham Ashton
Founder, The Agile Planner
http://www.theagileplanner.com | @agileplanner | @grahamashton