Eager loading has_many associations

[Angelim]

Hi guys,
I'm having trouble understanding what behavior I should expect from
eager loading has_many associations. Let's say we have a User and
Comment models. Naturally, a User has many Comments.
When I issue the following on console:
User.all.includes(:comments).each{|e| e.comments.inspect}

The mongo driver tells me it's performing the following queries:
['system.namespaces'].find({})
['users'].find({})
['system.namespaces'].find({})
['comments'].find({"user_id"=>{"$in"=>[BSON::ObjectId('4f2e761aa4cd5bcea700000a'),
BSON::ObjectId('4f2e764ca4cd5bcea7000013'),
BSON::ObjectId('4f28f0baa4cd5babb3000019'),
BSON::ObjectId('4f2b4725a4cd5b55c100000f'),
BSON::ObjectId('4f2bb90aa4cd5b601a000012')]}})

['comments'].find({"user_id"=>BSON::ObjectId('4f2e761aa4cd5bcea700000a')})
['comments'].find({"user_id"=>BSON::ObjectId('4f2e764ca4cd5bcea7000013')})
['comments'].find({"user_id"=>BSON::ObjectId('4f28f0baa4cd5babb3000019')})
['comments'].find({"user_id"=>BSON::ObjectId('4f2b4725a4cd5b55c100000f')})

Maybe I'm just confused on how this is supposed to work, but aren't
the last 4 queries exactly the ones we would be saving when eager
loading?

Thanks

Angelim

[Durran Jordan]

Do you have the identity map enabled?

2012/2/10 Angelim <ang...@gmail.com>

[Angelim]

Hi Duran,
yes, I can confirm that either on mongoid.yml, as in the same terminal
session where I've performed that test:
Mongoid.config.identity_map_enabled #=> true

I'm on mongoid-10660a2fdfdb, a little bit further than 2.4.3. I was
just wondering if this is the expected behavior. If not, I can regress
and try to find out where it started happening.
The funny thing is that those users are indeed in the identity map at
that point. Mongoid::IdentityMap.get(User,
BSON::ObjectId('4f2b4725a4cd5b55c100000f')) returns the right
document.

Thanks again

On Feb 10, 7:00 am, Durran Jordan <dur...@gmail.com> wrote:
> Do you have the identity map enabled?
>

> 2012/2/10 Angelim <ange...@gmail.com>

>
>
>
>
>
>
>
> > Hi guys,
> > I'm having trouble understanding what behavior I should expect from
> > eager loading has_many associations. Let's say we have a User and
> > Comment models. Naturally, a User has many Comments.
> > When I issue the following on console:
> > User.all.includes(:comments).each{|e| e.comments.inspect}
>
> > The mongo driver tells me it's performing the following queries:
> > ['system.namespaces'].find({})
> > ['users'].find({})
> > ['system.namespaces'].find({})
>

> > ['comments'].find({"user_id"=>{"$in"=>[BSON::ObjectId('4f2e761aa4cd5bcea700 000a'),

[Durran Jordan]

I'm wondering if it's just the #inspect call going through method_missing on the proxy and hitting the database again... What if you just print out a single field?

2012/2/10 Angelim <ang...@gmail.com>

[Angelim]

User.all.includes(:comments).each{|e| e.comments.map(&:body)} logs the
same thing.

On Feb 10, 1:53 pm, Durran Jordan <dur...@gmail.com> wrote:
> I'm wondering if it's just the #inspect call going through method_missing
> on the proxy and hitting the database again... What if you just print out a
> single field?
>

> 2012/2/10 Angelim <ange...@gmail.com>

[Durran Jordan]

And what about excluding the #all:

User.includes(:comments)

2012/2/10 Angelim <ang...@gmail.com>

[Angelim]

User.includes(:comments).each{|e| e.comments.map(&:body) logs the same
queries. I'll try different versions of mongoid in fresh projects and
report back during this weekend.
Thank you one more time.

On Feb 10, 6:54 pm, Durran Jordan <dur...@gmail.com> wrote:
> And what about excluding the #all:
>
> User.includes(:comments)
>

> 2012/2/10 Angelim <ange...@gmail.com>

[Durran Jordan]

Yeah I was just thinking of various things since it's working fine for me... Can you link to a gist with your Gemfile, Gemfile.lock, and User and Comments models?

2012/2/11 Angelim <ang...@gmail.com>

[Angelim]

Durran,
after some digging I came to the conclusion that we cannot consider it
a bug. Those individual queries to user are in respect of those that
don't have any entries, therefore, based on the first query, weren't
loaded into the map. I'm attaching a gist, like you asked, so you can
see some other logs.

https://gist.github.com/1799236

Do you think we can work around this to avoid those hits?

On Feb 11, 7:19 am, Durran Jordan <dur...@gmail.com> wrote:
> Yeah I was just thinking of various things since it's working fine for
> me... Can you link to a gist with your Gemfile, Gemfile.lock, and User and
> Comments models?
>

> 2012/2/11 Angelim <ange...@gmail.com>

[mmlac]

I would say it is a bug. And can become a major problem as well. In my example I have a set that either has issues or volumes.

So the expected case is: 

When I include volumes and issues, Mongoid should know that this nil is actually a database-nil and not a "I did not fetch it yet"-nil. (I assume that is how determining if a query is necessary works)

Foo.all.includes(:issues).includes(:volumes).each do |a|
p a.issues.try(:title)
p a.volumes.try(:title)

should hit the database only three times (get Foo.all, get all issues w/ $in, get all volumes w/ $in) and not worst-case 2n+3 times which on a set of 10.000 entries will generate 20.003 queries compared to 3 and a few 'get more'!


Furthermore I noticed that the count operation does also seem to trigger a database hit instead of using the eager-loaded data and running a Array.count on it - Not sure if that is intentional, b/c mongo might be faster, but bombarding the database with requests that already loaded data can perfectly solve seems counter-intuitive to me.

I would very much appreciate if there could be taken another look at this issue and possible solutions discussed.

Thanks,
Markus


p.s.: please tell me if I am wrong or there is an obvious solution that I seem to miss here  (except creating three queries that treat each case (none, only issues, only volumes) separately)