domain modeling

[Dan Croak]

I watched this video of Durran talking about Mongoid:

http://www.livestream.com/hashrocket/video?clipId=pla_686e2833-4950-4389-8c69-46776949da1e

He talked about using Mongo for hierarchical data and SQL for
relational data. I'm trying to wrap my head around what that means in
practice for a Rails app.

I've got an existing construction safety application. I'm spiking a
conversion of some of the domain model to use Mongoid. I was hoping
someone here could share some advice about which models are better
suited for Mongo and which are better suited for SQL.

Company
has many Projects

Project
has many Safety Inspectors (users)
has one Safety Manager (user)
has one Project Manager (user)
has many Zones
has many Incidents

Safety Category
has many Safety Issues

Incident
belongs to Project
belongs to Safety Issue
belongs to Safety Inspector

So I'm thinking that Safety Category/Issue is clearly hierarchical
(even displayed that way to users). More of a grey area for me is
whether it makes sense to convert Company/Project/Incident to Mongo
(that is a hierarchy but once signed in, you're basically only dealing
with Incidents) and whether it makes sense to convert the different
types of users to Mongo (there is a clear hierarchy of bosses and each
successive boss can see/do everything on the app that their direct
reports can).

Any advice would be greatly appreciated.

Thanks,

--
Dan Croak
@Croaky

[matthewford]

I too am also interested about how people go about domain modeling
with Mongoid.

With mongomapper I would have the User as a separate collection and
have an arrays of the id's for the different members of the project,
although I'm not sure if that's the best way to go about it with
mongoid.

On Jan 16, 7:56 pm, Dan Croak <dcr...@thoughtbot.com> wrote:
> I watched this video of Durran talking about Mongoid:
>

> http://www.livestream.com/hashrocket/video?clipId=pla_686e2833-4950-4...

[Durran Jordan]

Sorry I haven't skipped this on purpose yet - just busy and going through everything one at a time, I'll respond properly soon once I've examined the post more thoroughly.

2010/1/16 Dan Croak <dcr...@thoughtbot.com>

[Durran Jordan]

Sorry I ended up fixing a lot of issues today related to work and haven't been able to get on Skype and chat yet, however I did think about your domain last night a bit...

So first off we always store users in the relational database, especially since passwords are stored there for users and it's easy to get at Mongo's data from the filesystem without db access, and easy to parse it. So I would model the SafetyInspecor, SafetyManager, and ProjectManager classes in ActiveRecord. Zone seems like lookup data that wouldn't change often so I'd stick that in the relational database as well.

Category/SafetyIssue does fit into a hierarchy, as well as Company, Project and Incident from what I can gather...

So here's a gist of my first thoughts for this:

http://gist.github.com/280536

I typed it directly in there so pardon any typos.

Even though you stated you were only really dealing with Incidents once you log in, they still only exist in the context of a Project and Company - so essentially you'll make 1 query for the company and you already have all its projects and incidents. We've been using inherited_resources in our controllers to handle the nesting out of the box and it works great, however if you want to get experimental we are moving all our controllers over to use let-it-be so I'll post an update on how we handle that with Mongoid.

let-it-be is here: http://github.com/voxdolo/let-it-be

Hope this is a good start... If I can get finished up with work at a decent hour I'll hop on skype, otherwise it will have to be tomorrow if you want to discuss further.

d

2010/1/16 Dan Croak <dcr...@thoughtbot.com>

[matthewford]

Ah ok, so you are using more of a hybrid setup, is there any
documentation of how to setup mongoid, with a relational db (also can
you use something other than AR, like DM)?

I was hoping to migrate fully over to mongodb, although it's not a
dealbreaker. On a side note I tried to fork devise to add a Mongid
adapter

http://github.com/bitzesty/devise/tree/mongoid

However I get the following error when using :validatable
TypeError ([] is not a symbol):
/opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/durran-
validatable-2.0.1/lib/validations/validates_uniqueness_of.rb:8:in
`send'

It seems to be due to the scope being an empty array by default

Matt

> >http://www.livestream.com/hashrocket/video?clipId=pla_686e2833-4950-4...

[Dan Croak]

Thanks, Durran. This is helpful. I don't think we need to schedule a
Skype session but I have a couple of other questions if you'll indulge
me...

Images

I think I heard you say in the video you use CarrierWave? Looks cool,
I'm going to prefer to store images on S3 for this project. Each
Incident has many images (probably 2-5, as high-res as mobile phones
will allow). Does it make sense to store these as embedded documents
where their only values are links to S3 URLs?

Users

If passwords are the issue in Mongo, does it make sense to simply
store encrypted passwords in a relational db and move the User model
to Mongo if a domain model suggests it is necessary?

Lookup data

Can you explain a little further why lookup data is better suited for
SQL? You're right that Zone qualifies as such.

Controllers

I'm very familiar with Inherited Resources from our last few projects
and can see how it might fit. Let-It-Be looks interesting. I've always
had a beef with copying over instance variables. Will keep an eye on
it, thanks.

Thanks again, Durran. Really appreciate your time. Will report back if
we push this app into production with any of it using Mongo. I like
the Mongoid/Hashrocket/Pharma app story. Jives well with the
cleanliness and professionalism of the Mongoid source. Maybe
Mongoid/thoughtbot/construction safety story could bolster that?

--
Dan Croak
@Croaky

[Kyle Banker]

Durran,

Can you talk more about your reluctance to store passwords in MongoDB?  Assuming that you're properly encrypting the passwords, and that the data files have the proper permissions set, I'm not seeing how a relational database would be any more secure.

Kyle

[Durran Jordan]

You are right in that my statement about security is incorrect given proper precautions are taken, so I'll retract that comment. :)

2010/1/19 Kyle Banker <ky...@10gen.com>

[Durran Jordan]

2010/1/19 Dan Croak <dcr...@thoughtbot.com>

Thanks, Durran. This is helpful. I don't think we need to schedule a
Skype session but I have a couple of other questions if you'll indulge
me...

Images

I think I heard you say in the video you use CarrierWave? Looks cool,
I'm going to prefer to store images on S3 for this project. Each
Incident has many images (probably 2-5, as high-res as mobile phones
will allow). Does it make sense to store these as embedded documents
where their only values are links to S3 URLs?

That would be the approach i'd take - also note that carrierwave does support gridfs storage in mongo if you want look into that.
 

Users

If passwords are the issue in Mongo, does it make sense to simply
store encrypted passwords in a relational db and move the User model
to Mongo if a domain model suggests it is necessary?

Given Kyle's comment he's actually right... I'll just need to add support for Mongoid in a few of the gems out there to make this easier... Devise will probably be the first but I think someone was already working on that.
 

Lookup data

Can you explain a little further why lookup data is better suited for
SQL? You're right that Zone qualifies as such.

For us it just felt like a more natural fit. We use Mongo for a single purpose - performance when dealing with a large hierarchical domain model that requires the entire object graph to be loaded in order to do the application work. In our case our lookup data is in fact relational (consider drugs, ingredients, dosage forms, packaging, etc) and also provided by a third party. It didn't make sense for us to copy it into Mongo just to do so. In your case it's just one model so I could go either way.

One thing to note, and this may or may not apply to you given the size of your data set. Our Mongo database's physical storage currently takes up 60GB where the MySQL copy of the data is 10GB (Obviously rounded). We do imports and exports between us and the legacy etl team so we can see it side by side right now. If you're using something like ey cloud or amazon where storage is expensive and you're on a budget so also may want to think about those implications... :)
 

Controllers

I'm very familiar with Inherited Resources from our last few projects
and can see how it might fit. Let-It-Be looks interesting. I've always
had a beef with copying over instance variables. Will keep an eye on
it, thanks.

Thanks again, Durran. Really appreciate your time. Will report back if
we push this app into production with any of it using Mongo. I like
the Mongoid/Hashrocket/Pharma app story. Jives well with the
cleanliness and professionalism of the Mongoid source. Maybe
Mongoid/thoughtbot/construction safety story could bolster that?

I hope so. :) Mongo's been nothing but fun so far.
 

[Dan Croak]

On Tue, Jan 19, 2010 at 5:35 PM, Durran Jordan <dur...@gmail.com> wrote:

> Given Kyle's comment he's actually right... I'll just need to add support
> for Mongoid in a few of the gems out there to make this easier... Devise
> will probably be the first but I think someone was already working on that.

Mongoid can be used with Clearance right now without problem:

class User
include Mongoid::Document
include Clearance::User

field :email
field :encrypted_password
field :salt
field :confirmation_token
field :remember_token
field :email_confirmed, :type => Boolean, :default => false
end

--
Dan Croak
@Croaky

[Scott Weeks]

It mostly does although it seems that I had a problem with the
validations since the validatable gem doesn't accept
the :case_sensitive option for validates_uniqueness_of.

[Dan Croak]

Ah. So that would require a fork of validatable or need to wait until
ActiveModel::Validations are used?

I don't think there's anything else in the module that Mongoid can't handle:

http://github.com/thoughtbot/clearance/blob/master/lib/clearance/user.rb

Dynamic finders (find_by_email) are cool, right?

--
Dan Croak
@Croaky

[Scott Weeks]

Curses, I forgot about that. I hacked it in (just added a class
method) This is what my User class looks like

class User
include Mongoid::Document

include Mongoid::Timestamps

extend Clearance::User::ClassMethods
include Clearance::User::InstanceMethods
include Clearance::User::AttrAccessor
include Clearance::User::Callbacks

.# . . manually added validations and fields

def self.find_by_email(email)
self.first :conditions => { :email => email }
end
end

I'm planning on adding Clearance to another class in my app and was
waiting until then to pull that stuff out into its own module.

[HP]

There is a Mongoid fork of Devise waiting to be merged in:
http://github.com/plataformatec/devise/issues#issue/85 Needs help
testing it.

- HP

On Jan 20, 7:49 pm, Scott Weeks <scott.we...@gmail.com> wrote:
> Curses, I forgot about that. I hacked it in (just added a class  
> method) This is what my User class looks like
>
> class User
>    include Mongoid::Document
>    include Mongoid::Timestamps
>
>    extend  Clearance::User::ClassMethods
>    include Clearance::User::InstanceMethods
>    include Clearance::User::AttrAccessor
>    include Clearance::User::Callbacks
>
>   .#  . . manually added validations and fields
>
>    def self.find_by_email(email)
>      self.first :conditions => { :email => email }
>    end
> end
>
> I'm planning on adding Clearance to another class in my app and was  
> waiting until then to pull that stuff out into its own module.
>
> On Jan 20, 2010, at 1:36 PM, Dan Croak wrote:
>
>
>
> > Ah. So that would require a fork of validatable or need to wait until
> > ActiveModel::Validations are used?
>
> > I don't think there's anything else in the module that Mongoid can't  
> > handle:
>
> >http://github.com/thoughtbot/clearance/blob/master/lib/clearance/user.rb
>
> > Dynamic finders (find_by_email) are cool, right?
>

> > On Wed, Jan 20, 2010 at 1:33 PM, Scott Weeks <scott.we...@gmail.com>  

[Durran Jordan]

No they actually aren't. (They used to be) :)

I removed the dynamic finders in favor of datamapper's approach, which adds 2 extra methods: find_or_create_by and find_or_initialize_by - both which just take a hash of attributes... All other queries should be going through the criteria api.

This was done so method_missing could be used in a cleaner way to handle dynamic attributes on an instance by instance basis.

2010/1/20 Dan Croak <dcr...@thoughtbot.com>

[matthewford]

Right, my fork has one major issue, the session isn't persisted for
some reason after authentication - it's a bit of a blocker, but I'll
take another look at it tomorrow.

On Jan 20, 7:22 pm, HP <hanspet...@gmail.com> wrote:
> There is a Mongoid fork of Devise waiting to be merged in:http://github.com/plataformatec/devise/issues#issue/85Needs help

[Durran Jordan]

I'll have a look at it tonight as well and see if I can help...

2010/1/20 matthewford <ma...@bitzesty.com>

[Durran Jordan]

I'll also update the validatable gem for that... If you want to live on the edge however you can grab the Mongoid prerelease branch and build from there - it's using ActiveModel::Validations and is Rails 3 compatible at the current point, unless something major changed in the last few days...

2010/1/20 Dan Croak <dcr...@thoughtbot.com>

[HP]

> Right, my fork has one major issue, the session isn't persisted for
> some reason after authentication - it's a bit of a blocker, but I'll
> take another look at it tomorrow.

Try adding this to an initializer (config/initializer/warden.rb):

class Warden::SessionSerializer
def serialize(user)
user.id
end

def deserialize(id)
User.criteria.id(id).first
end
end

The default/built-in Warden::SessionSerializer will not work with
Mongoid. I'm using RailsWarden, and the above works.

- HP

Devise serializer:
class Warden::SessionSerializer
def serialize(record)
[record.class, record.id]
end

def deserialize(keys)
klass, id = keys
klass.find(:first, :conditions => { :id => id })
end
end