Best way to handle server-side aggregates.

[Tyler Eaves]

I am evaluating the use of Meteor for an in-house project revolving around analytics - the kind of thing that should be MongoDB's bread-and-butter. Unfortunately, for this to work I need to be able to run aggregate queries, which Meteor is currently incapable of doing, and the Meteor plugins that claim to extend this functionality are old and non-functional. 

So, I tried doing the query server side using the nodejs native mongo driver. This works great - but I can't figure out how I'm supposed to actually publish this to the client. I tried wrapping it as a Meteor.method, and this also seems to work, near as I can tell, but I can't subscribe to it and have it work. I do _not_ need reactive functionality with this but I do need some way of handling it.

[Andrew Mao]

http://stackoverflow.com/a/18884223/586086

[Abigail Watson]

Depends a bit on how you want to do your aggregation, obviously.  Andrew's recipe works good if you have a smallish sized database, and want to get up-to-the-minute aggregation stats each time you connect.  For instance, it works well for aggregating stats for logged in client in a chat program.

For other types of apps, there are some other patterns.  We're finding that with larger databases that have streaming data coming in, we needed a statistics writer to write data into a collection specifically designed for handling stats.  We use an observeChanges query, and simply aggregate stats as the records come into the database.  Basically, a real-time aggregation.  

We're also finding that it makes a lot of sense to run the aggregation pipeline that Andrew demonstrates from an API call, or a Meteor Method, rather than in the publication function.  Basically, it's asynchronous enough and expensive enough that it makes sense to fire it with a Meteor.call(), rather than put it into a publication function.  If you've got it firing as a Meteor.method(), you simply need to return a json object, and then catch it in the Meteor.call() callback.   

Meteor.call('aggregateStats', function(error, result){

  if(result){

    console.log(result);

  }  

});

However, I prefer to have a DailyStats collection, or something similar, and to write the aggregated statistics into a single record in the collection.  Then you get an easy timeline of statistics, and the publication/subscription is nothing more complicated than what's widely documented in all the tutorials and best practices.  But, again, that approach requires a statistics writer file or function.  

[Andrew Mao]

Good writeup, Abigail. I'm going to link to it from my answer.

[David Collier]

hi -

However, I prefer to have a DailyStats collection, or something similar, and to write the aggregated statistics into a single record in the collection.  Then you get an easy timeline of statistics, and the publication/subscription is nothing more complicated than what's widely documented in all the tutorials and best practices.  But, again, that approach requires a statistics writer file or function.  

this is the approach i am planning to take for something. 

how do you generally fire off these kind of background tasks with a meteor app? there seem to be some cron like packages on atmosphere but wondering what is recommended. some of them basically just seem to be using setInterval internally.

also, is mongo aggregation pipeline going to be supported within meteor directly, or will it be required to dig into node/native libraries?

tx!

[matt debergalis]

My understanding is Mongo 2.6 supports returning aggregate queries as
cursors. If so, it'd be utterly reasonable to publish server-side
aggregate queries. Not sure how much node driver work would have to
come first. It also supports directly writing the output of an
aggregate query into another collection.

Adding the aggregate pipeline to minimongo for use on the client is
another matter but shouldn't be necessary for the dashboard use case.

> --
> You received this message because you are subscribed to the Google Groups
> "meteor-talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to meteor-talk...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Arunoda Susiripala]

Yep 2.6 supports returning a cursor. The point of that is to allow large aggregation result. Earlier it was limited to something like 16 MB (not sure about the exact limit).

My Questions are, 

  *are we going to make aggregations reactive? 

  *allow to re-run them when something changes? 

  *any chance we can use oplog to detect the changes and re-run aggregations locally?

BTW: Adding mini-mongo support would be very useful.

---

Arunoda Susiripala

I curate Meteor Weekly - Check it out!