[Proposal] Denormalizing your Rails application

[Daniel Lucraft]

Hi all,

*** Denormalizing your Rails application

At Songkick we have spent the past few weeks developing a pattern for
denormalizing our relational data to improve our application
performance. (We haven't deployed much of this work yet, but will have
by the time the Manor comes around.)

In this talk we'll cover the architecture of two as-yet-unreleased
Ruby gems we are using internally:

1. asynchronous_observers: This extracts the Rails Observers into a
separate daemon, with CRUD operations on your models being sent over
AMQP.

2. silos: This is a MongoDB wrapper that permits simple memoization of
methods on specific Ruby objects into the database. It also has test
helpers that will auto-check your update scheme for you, which is
nifty.

Taken together these make adding new denormalizations as close to
trivial as we think possible. Since this is a pretty experimental
system, I'd also be interested in feedback after the session.

thanks,
Dan

___________________________________
Daniel Lucraft

danlucraft.com/blog
twitter.com/danlucraft

[Daniel Lucraft]

BTW, I definitely typed "[Proposal]" in the subject line, but Google
Groups web interface stripped it out. What is that about?

D

[Matt Patterson]

It (gmail) uses [token] in the subject line for tagging/filing. Was in
subject line I received...

Sent from my iPhone

> --
> http://rubymanor.org
> Group home: http://groups.google.com/group/ruby-manor?hl=en
> Unsubscribe at ruby-manor+...@googlegroups.com

[Roland Swingler]

This sounds like it could be interesting.

Some specific things I'd be interested in:

* what led you to mongodb rather than some other NoSQL db? There seems
to be a "paradox of choice" around NoSQL dbs at the moment, and it
would be interesting to hear good reasons for choosing one rather than
another.
* presumably you can only memoize "pure" methods - have you come up
with any strategies/conventions for ensuring that? What does
serializing memoized results rather than just keeping them in memory
buy you?
* are your model observers running in or out of transaction
boundaries? We've run into problems where queued tasks run from a
standard after_create method are actually too fast - and start running
before the transaction has been commited.

Cheers,
Roland

[Alan Kennedy]

+1. It would be interesting to hear the criteria you used to denormalise the data and the migration strategy to move from sql to MongoDB.

Alan

2009/11/25 Roland Swingler <roland....@gmail.com>

[Jonathan Conway]

+1 sounds like an interesting subject

---------------------
www.jaikoo.com

[Matt Patterson]

+1 for me - really fascinated by the whole denormalising-via-almost-caching document DB thing...

Matt

> --
> http://rubymanor.org
> Group home: http://groups.google.com/group/ruby-manor?hl=en
> Unsubscribe at ruby-manor+...@googlegroups.com



--

Matt Patterson | Design & Code
<ma...@reprocessed.org> | http://www.reprocessed.org/

[Murray Steele]

The answer might be a simple "no" but I'd be interested to hear if
there are any gotchas with testing stuff that you've denormalised.
(That's me assuming it's some sort of transparent layer, like
caching... it may not be).

Muz


2009/11/25 Matt Patterson <ma...@reprocessed.org>:

[Matt Patterson]

On 25 Nov 2009, at 11:44, Murray Steele wrote:

> The answer might be a simple "no" but I'd be interested to hear if
> there are any gotchas with testing stuff that you've denormalised.
> (That's me assuming it's some sort of transparent layer, like
> caching... it may not be).

This is what I'm really keen to find out - because denormalising used to be Gotcha the Barbarian.

Matt

[Daniel Lucraft]

No reason we can't cover all the points mentioned. I can briefly explain the benefits of Mongo, and there'll be example code (for tests too) to show how it works.

Roland: you may find this plugin useful: http://github.com/GUI/after_commit Make sure to look in the network for that as there are lots of more recent repos.

best,
Dan

_______________________________
Daniel Lucraft

twitter.com/danlucraft
danlucraft.com/blog

[Matt Southerden]

+1

I'd like to hear about this, Dan.

Matt.