Meteor Mongo query performance problem

[Kostya Marchenko]

I was doing some performance benchmarks for Meteor recently to select an approach how to process large amount of data (450000 records) and found weird thing – Meteor was very slow in querying data.

I had two collections: tests and testResults. Tests collection has 15000 records and test results has 450000 in total, 30 for each test. Each test result has key with testId to reference correct test.

So on server side I was iterating over all tests in test collection and then inside of that loop querying test results collection to get all test results for that test.

The operation was so slow that after waiting 5 minutes I had to kill the process. Doing exactly the same operation in Ruby with Mongoid driver was taking about 29 seconds.

Sample code that I used:

Meteor.startup(function () {         

    

    var start = new Date();

        

    var counter = 0;

    Tests.find({}).forEach(function(test) {            

        TestResults.find({testCaseId: test.testCaseId}).forEach(function(testResult) {                             

            counter++;

        });                

    })

    

    var time = new Date() - start;

    console.log("Count: "+counter);

    console.log("Took: "+time+"ms");

  });

Am I doing something wrong or there's a problem with Meteor's mongo driver?

I also tried fetching all records as arrays and then using underscore's where() function to query test results – seems to be similar performance.

Thanks,

Kostya

[Kostya Marchenko]

Another problem I found is related to livedata package – looking at it's source here https://github.com/meteor/meteor/blob/master/packages/mongo-livedata/mongo_driver.js it looks like it polls DB every 10 seconds for every query that you have in your app active at the moment, retrieves all results and diffs them against those that already displayed.

That results in massive usage of resources all the time even when it's idle:

 

[matt debergalis]

That shouldn't be. I'm pretty sure you're missing an index on
testResults.testCaseId. There's no Meteor API yet to create an index,
but you can do something like this:

[i386:fresh-pond-parkway!deberg] meteor mongo
MongoDB shell version: 2.2.1
connecting to: 127.0.0.1:3002/meteor
> db.testResults.ensureIndex({testCaseId: 1});

On my laptop, your code runs in about 10 seconds against an indexed DB
w/ 450k records. (That'll vary w/ document size, though. I don't
know how big your records are.)

cheers,
matt

> --
>
>

[matt debergalis]

This polling is so that Meteor will notice DB changes that didn't come through the Meteor server process.

Many apps don't need this, though.  We're considering ways to disable it.  We also have a more efficient implementation in the hopper.

--
 
 

[Owen Rees-Hayward]

Hi Matt,

Is there a rough ETA on the 'more efficient implementation' that's in the hopper? The poll delay comes into play if you need to do any incremental map-reduce work directly on the mongo instance. 

It would be great if the delay could be eliminated/significantly reduced; in every other way the reactive model in Meteor is outstanding. 

Cheers, Owen

[matt debergalis]

No ETA yet.  We're still getting the pieces in place -- Galaxy paves the way for this and many other scaling items.

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Owen Rees-Hayward]

Thanks for the update Matt.

[Owen Rees-Hayward]

Matt,

Just to clarify. Are you saying this issue is not likely to be resolved int the Meteor core open-source release, but only in the Galaxy product?

Cheers, Owen

On Wednesday, 26 June 2013 23:09:47 UTC+1, Matt DeBergalis wrote:

[Arunoda Susiripala]

I think those will be added to the core. I heard it somewhere else. May be on a DiscoverMeteor interview.

So no worries :)

--
Arunoda Susiripala

@arunoda

https://github.com/arunoda

http://www.linkedin.com/in/arunoda

[matt debergalis]

One way or another, 1.0 will have a more efficient mechanism for reacting to database changes, so that updates don't trigger system-wide recalculations.  I'm not certain if 1.0 will support realtime observes on arbitrary updates from outside the Meteor server.  But it should be easy to route those updates through Meteor.

[Andrew Mao]

I vague remember seeing ensureIndex in someone's JS code in a Meteor project. Is there any way to create an index from the server instead of having to use the Mongo console?

[Owen Rees-Hayward]

Thanks for the update Matt.

[Arunoda Susiripala]

I started working on a better mongodb implementation and here comes the first version.

I names it has Smart Collections

I does not poll mongo, check it out.