Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Creating 'summary documents' on active data
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  4 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Wouter  
View profile  
 More options Oct 14 2009, 10:34 am
From: Wouter <wiegm...@gmail.com>
Date: Wed, 14 Oct 2009 07:34:40 -0700 (PDT)
Local: Wed, Oct 14 2009 10:34 am
Subject: Creating 'summary documents' on active data
I want to be able to create 'summary documents' (the Mongo equivalent
of summary tables :-)).

Once the summary document is created, it will be updated using $inc /
$set, meaning the summary will be always up to date (that's what I
need :-)).

The process to create a summary looks like this:

1) Retrieve relevant data

2) Calculate summary

3) Store results in summary document

4) Update summary document when data is updated/added.

However, if data is added/updated between step 1 and step 3, the
summary will be incorrect. Any suggestions how to deal with this?!


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
dwight_10gen  
View profile  
 More options Oct 14 2009, 10:51 am
From: dwight_10gen <dwi...@10gen.com>
Date: Wed, 14 Oct 2009 07:51:23 -0700 (PDT)
Local: Wed, Oct 14 2009 10:51 am
Subject: Re: Creating 'summary documents' on active data
how often do you want to (re)calculate the summarizes?  near realtime
or infrequent (daily)?

On Oct 14, 10:34 am, Wouter <wiegm...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Wouter  
View profile  
 More options Oct 14 2009, 11:34 am
From: Wouter <wiegm...@gmail.com>
Date: Wed, 14 Oct 2009 08:34:36 -0700 (PDT)
Local: Wed, Oct 14 2009 11:34 am
Subject: Re: Creating 'summary documents' on active data
The summaries are never recalculated. Once a summary has been created,
it is updated in real time using $inc. I need real time summaries.

Let's say I'm offering real time analytics using Mongo (http://
blog.mongodb.org/post/171353301/using-mongodb-for-real-time-
analytics), but I also save info about each visit, and I allow
customized reports.

So with every visit:

1) Upsert data for this visitor in the visits collection.
2) Update all reports to which this visitor belongs (using $inc)

Initially, there is one report (summary document): the one that
includes all visitors to the site.

Now, users can create their own customized reports. EG: generate a
report for all US visitors that spend at least 5 minutes on my site.

1) Retrieve all visit data find({ site_id : 123, country : 'US',
total_time : { $gt : '300' }})

2) Calculate summary (number of visits, time_spent, other statistics)

3) Store results in summary document (db.reports.insert ({ ...}))

4) Update summary document when data is updated/added.
(db.reports.update( { total : $inc .... })

Now, if a US visitor is active on the site between step 1 and step 3,
the report could end up being incorrect...

How to prevent that :-)

On 14 okt, 16:51, dwight_10gen <dwi...@10gen.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mathias Stearn  
View profile  
 More options Oct 14 2009, 12:29 pm
From: Mathias Stearn <math...@10gen.com>
Date: Wed, 14 Oct 2009 12:29:27 -0400
Local: Wed, Oct 14 2009 12:29 pm
Subject: Re: Creating 'summary documents' on active data

One way would be to start the $inc process for new documents before you
start calculating the report, then rather than inserting the report
wholesale, you could upsert it using $inc on each field. You would have to
limit the timestamps in your find() when calculating the summary to ensure
that new visits aren't double counted.
--Mathias


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »