Help test minimongo improvements (for oplog!)

234 views
Skip to first unread message

David Glasser

unread,
Jan 3, 2014, 2:53:36 PM1/3/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Happy New Year Meteorites!

Right now I'm working on getting oplog to support as many queries as
possible. In preparation, I've given minimongo a thorough
refactoring, based on testing lots of weird MongoDB query edge cases.
That's because the oplog feature uses minimongo's selector logic on
the server in security-critical codepaths.

minimongo has pretty good test coverage, so I feel pretty confident
about these improvements. My belief is that they don't affect most
apps and make some others slightly more MongoDB-compatible, and that
they don't significantly affect performance. But maybe I missed
something. I'd love to get feedback from our users as to how these
changes affect your apps!

Just run meteor with "meteor --release mongo-refactor" and let me know
if you notice any differences.

(This is based on devel, which does not contain the 'shark'
template-engine changes.)

--dave

Tim Heckel

unread,
Jan 6, 2014, 1:52:23 PM1/6/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
David, I pulled down devel just recently and am running off of that...also, I had our mongo provider spin up a new replica set for the sake of testing the oplog tailing. I mentioned to Avi that we use a number of $-based queries...as those get added, can I assume that the FACTS observe-drivers-polling will go down, and observe-drivers-oplog will go up? I will run the app through its paces and let you know, but I have confirmed oplog tailing is working when I direct my local MONGO_URL and MONGO_OPLOG_URL to our testing databases in the cloud via the facts package. I sent you an email with a screenshot...

In the meantime, I'll use devel against our app for a bit here and let you know if I see any strange minimongo exceptions. Thanks!

David Glasser

unread,
Jan 6, 2014, 1:56:39 PM1/6/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Note that I haven't actually switched the $ queries over to oplog
observe driver yet, though I'm going to do that very soon. Merely
rewrote the underlying logic in minimongo itself (which will mostly
affect, or hopefully not affect, your *client app* right now).
> --
> You received this message because you are subscribed to the Google Groups
> "meteor-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to meteor-core...@googlegroups.com.
> To post to this group, send email to meteo...@googlegroups.com.
> Visit this group at http://groups.google.com/group/meteor-core.
> For more options, visit https://groups.google.com/groups/opt_out.

Tim Heckel

unread,
Jan 6, 2014, 3:15:15 PM1/6/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Right...that I understood...and so far so good. The client side queries all seem to be working well for me using the latest devel...hope this helps

David Glasser

unread,
Jan 8, 2014, 2:13:37 AM1/8/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Since this seems good so far, I've put out another release that enable
oplog for most $ operators!

Try your app with `meteor --release oplog-with-operators`! (In
production, you still need to configure the oplog environment
variable.)

oplog is still disabled for skip/limit queries, or for queries with
$where or $near.

Tim Heckel

unread,
Jan 8, 2014, 4:54:39 PM1/8/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
David, I am going to test this shortly -- thanks for these improvements, that sounds great! 

I have a question -- a bit off this topic. We are having some issues with our node servers cumulatively spiking since moving to 0.7.1 from 0.6.4.1....we have a sharded db, but have NOT specified MONGO_OPLOG_URL -- can you think of any difference adding the disable-oplog package would make if we haven't even specified an oplog url? I know sharding is not supported for oplog-tailing. I'm just trying to eliminate this as a possible issue for our prod env and the slowdown of the web servers. Does my question make sense?

Tim Heckel

unread,
Jan 9, 2014, 10:02:33 AM1/9/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
David, I just figured out our issue -- and it is NOT due to any issues with the mongo driver in 0.7.0.1 at all -- we had a problem with our main nginx configuration that was routing traffic with invalid sticky sessions to one of our web servers and causing havoc on that box. I did roll back to 0.6.6 instead of 0.6.4.1 because of a compatibility issue between iron-router and 0.6.4.1, but plan on going back to 0.7.0.1 now shortly. We will continue testing oplog tailing on our newly created replica set. Thanks again for your help in troubleshooting.

David Glasser

unread,
Jan 9, 2014, 1:14:09 PM1/9/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Great.

0.7.0.* is expected to be pretty compatible with 0.6.6.* apps, btw
(eg, we didn't think any changes were backwards-incompatible enough to
warrant release notes on 'meteor update').
> --
> You received this message because you are subscribed to the Google Groups
> "meteor-talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to meteor-talk...@googlegroups.com.

Adrian Lanning

unread,
Jan 11, 2014, 3:55:47 PM1/11/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Hi Dave,

Just ran a couple tests of our app using the oplog-with-operations release and everything looks good so far.

Noticed `observeChanges` now requires `_id` to be returned (ie. can't do `_id: 0` in `fields:{}`) so that would be nice to mention in the changelog.

Nice job!

David Glasser

unread,
Jan 11, 2014, 5:00:21 PM1/11/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Yeah, that was an explicit change which will show up in our
pre-release History.md pass. (I haven't been so good lately about
making History.md changes as I go; should try harder since I know how
important changelogs are!)

Tim Heckel

unread,
Jan 13, 2014, 4:38:55 PM1/13/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
David, I've finally had a chance to now run again against our oplog db....in the facts package, I no longer see any observe-drivers-polling, and I only see observe-drivers-oplog (there are 54) ... can I assume that we are now only using the oplog for difs? If so, woo-hoo ! We need to start doing some perf benchmarking really soon...

I will test our app with this setup and let you know

David Glasser

unread,
Jan 13, 2014, 4:53:22 PM1/13/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Yup, that's what that means.
> "meteor-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to meteor-core...@googlegroups.com.
> To post to this group, send email to meteo...@googlegroups.com.
>
> Visit this group at http://groups.google.com/group/meteor-core.

Nick Martin

unread,
Jan 13, 2014, 5:00:28 PM1/13/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
Yup, that means you're using oplog entirely. Very cool!

Are your running multiple meteor app processes? Depending on your data access pattern, oplog may or may not help on one CPU. Where it really shines is running on multiple CPUs at once.

Very curious to hear what you find =)
-- Nick

Tim Heckel

unread,
Jan 14, 2014, 6:39:48 PM1/14/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
I will hopefully have a chance to do some real testing very soon! I'm really excited to try this. And yes, we've had up to 12 (!) 8 core machines running our meteor app (each core we have configured listening on a different port with nginx) to keep up with the load. We've also recently (i think) been running up against connection pooling issues, and hope that the oplog tailing will help alleviate that as well...my biggest concern with the switch is our db performance, because we'll be going from a sharded to a non-sharded. Do you think it's beneficial then to change our readPreference to always be the secondary and leave the primary for the writes?

David Glasser

unread,
Jan 14, 2014, 7:19:24 PM1/14/14
to meteo...@googlegroups.com, meteo...@googlegroups.com
If that's consistent with the level of consistency your app requires.
Reply all
Reply to author
Forward
0 new messages