Re: [mongodb-user] Tail the oplog.rs, avoid full scann

[Bernie Hackett]

The OplogReplay flag only works if you query on the 'ts' field. If not, the server returns the error "no ts field in query". Here's an example from python:

>>> cur = c.local.oplog.rs.find().add_option(8)

>>> cur.next()

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "pymongo/cursor.py", line 814, in next

    if len(self.__data) or self._refresh():

  File "pymongo/cursor.py", line 763, in _refresh

    self.__uuid_subtype))

  File "pymongo/cursor.py", line 720, in __send_message

    self.__uuid_subtype)

  File "pymongo/helpers.py", line 100, in _unpack_response

    error_object["$err"])

pymongo.errors.OperationFailure: database error: no ts field in query

I believe if you do query on 'ts' with OplogReplay, all other fields in the query are ignored. OplogReplay tells the server to skip forward through the oplog to quickly find the Timestamp in question, instead of doing a regular table scan.

On Fri, May 17, 2013 at 11:09 AM, <kiw...@gmail.com> wrote:

Hi,
On a C++ program I tail the oplog.rs to read the new operations on a collection, using QueryOption_CursorTailable | QueryOption_AwaitData.
I filter the documents on "ns", "op", and "o._id" (my collection has a o._id).

I'd like to avoid the full rescan when a new connexion is needed. Adding QueryOption_OplogReplay with a filter on "ts" seems to be the standard solution, but I don't find many information on this special option OplogReplay.
I understand this is for internal use for the replication and it may disappear in the future (but I suppose the replication mechanism will not move a lot).

What does OplogReplay do exactly?
It seems to optimize the search for starting the cursor using "ts", but what is the complexity of this operation?
And does it work with other filters on the query like I do? (I suppose replication does not filter anything and reads all the operations on oplog.rs, so maybe this property is used in the implementation of OplogReplay).

Thanks

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
 
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[kiw...@gmail.com]

Thanks for the reply.


I tested if the other fields of the query are ignored or not: They are *not* ignored, at least on mongodb 2.4.3.

I searched on the mognodb code where it could be used, and I concluded I'm not familiar enough with the code to conclude anything.
In FindingStartCursor::init() only the "ts" field of the query is used, this seems to be the only usage of _qp._originalQuery in FindingStartCursor.
But in QueryPlan we have other usages of it, so maybe the filtering is done here.

Is this property durable though?


Also, cursor->more() returns every 5 secondes when there is nothing more. Is this timeout configurable somewhere?

[Bernie Hackett]

The oplog is a capped collection with a natural order that doesn't change (you can't update documents in a capped collection such that their position in the collection changes). The normal replication cursor only needs to keep track of the value of 'ts'. Your tailable cursor should be the same, unless I misunderstand what you are trying to do.

> Is this property durable though?

What property?

> Also, cursor->more() returns every 5 secondes when there is nothing more. Is this timeout configurable somewhere?

I don't think so.

[kiw...@gmail.com]

Le mardi 21 mai 2013 16:08:16 UTC+2, Bernie Hackett a écrit :

The oplog is a capped collection with a natural order that doesn't change (you can't update documents in a capped collection such that their position in the collection changes). The normal replication cursor only needs to keep track of the value of 'ts'. Your tailable cursor should be the same, unless I misunderstand what you are trying to do.

I am trying to tail the oplog.rs this way:
1/ I don't want to start from the beginning: I query with 'ts': {$gt: startts} and QueryOption_CursorTailable)
2/ I don't want all the operations: I query with op: "i" and ns: "mydb.mycollection"
 
This seems to work: the point 2/ is respected: if I do a delete on mydb.mycollection the client never receives the operation via the cursor.

But this contradicts what you said previously: you said the fields other than 'ts' are ignored with QueryOption_CursorTailable.

> Is this property durable though?

What property?

The property == The fact that the fields other than 'ts' are *not* ignored with QueryOption_CursorTailable.

If this property of mongodb is true only by accident then I should probably not rely on this in my code.

I understand the replication does not use this property to work as it only requires the 'ts' field to work. So this property may become false in the future without breaking the replication. So this probably makes this property less durable.
 

> Also, cursor->more() returns every 5 secondes when there is nothing more. Is this timeout configurable somewhere?

I don't think so.

OK.