Pymongo no_cursor_timeout ignored

404 views
Skip to first unread message

Anthony Leung

unread,
Apr 15, 2016, 11:11:28 AM4/15/16
to mongod...@googlegroups.com

There are about 350,000 records in the below test, which iterate just fine with a simple find and no wait inbetween, but if you introduce a long wait of the order of 15 minutes, the cursor becomes expired after the long wait, even with the no_cursor_timeout parameter set to True.

How can one keep the cursor open?

Minimum case:

import pymongo
import time

d = pymongo.Connection()['c']['c']
out = []
for a in d.find(no_cursor_timeout=True):
    out.append(a['data'][0:12])
    if len(out) == 1000:      # Run through at least 500 results first
        time.sleep(60 * 18)   # Wait 18 minutes
    if len(out) % 500 == 0:   # Continue printing status every 500 documents
        print len(out)

Output:

:~/calculators/dataenrich$ python test.py
500

1000
Traceback (most recent call last):
  File "test.py", line 29, in <module>
    for a in d3.find(no_cursor_timeout=True):
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1076, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1037, in _refresh
    limit, self.__id))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 958, in __send_message
    self.__compile_re)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/helpers.py", line 101, in _unpack_response
    cursor_id)
pymongo.errors.CursorNotFound: cursor id '94954022856' not valid at server

On Apr 15, 2016 4:06 PM, <mongod...@googlegroups.com> wrote:
"Szaniszlo Szöke" <szoke.s...@gmail.com>: Apr 15 07:13AM -0700

Hi everyone,
 
I'm trying to get the opposite of a complex search using $not.
Let's take a very simple case.
Searching for: (_id == 5) || (_id == 9) is done with command:
{ $or: [{_id: 5}, {_id: 9}]}
 
I expected to find the opposite by writing:
{$not: { $or: [{_id: 5}, {_id: 9}]}}
 
but that doesn't work, the correct syntax being:
{$and: [{_id: {$ne: 5}}, {_id: {$ne: 9}}]}
 
in other words: *is there a way to reverse a query, like in SQL, by using a
NOT command in front of it ?*
 
Note: the real query I have to reverse is a mix of AND, OR, REGEX, etc. so
I have no way to parse it and reverse it "by hand".
 
For those interested, the equivalent C++ code looks like:
mongo::BSONObj obj1 = BSON("_id" << BSON("$eq" << 5));
mongo::BSONObj obj2 = BSON("_id" << BSON("$eq" << 9));
mongo::BSONObj obj12 = mongo::OR(obj1, obj2);
 
// unfortunately, this doesn't work:
mongo::BSONObj obj = BSON("&not" << obj12);
 
Thanks in advance for any help.
Asya Kamsky <as...@mongodb.com>: Apr 15 10:38AM -0400

Do you mean the $nor
<https://docs.mongodb.org/manual/reference/operator/query/nor/> operator?
 
 
On Fri, Apr 15, 2016 at 10:13 AM, Szaniszlo Szöke <szoke.s...@gmail.com
 
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
"Szaniszlo Szöke" <szoke.s...@gmail.com>: Apr 15 07:42AM -0700

No, I mean the *$not* operator
"Szaniszlo Szöke" <szoke.s...@gmail.com>: Apr 15 07:46AM -0700

In SQL, this query:
select _id, name from table_0 where (_id = 5) || (_id = 9) ;
 
becomes this query:
select _id, name from table_0 where not((_id = 5) || (_id = 9)) ;
 
how can I do that with MongoDB ?
John Reno <jlr...@gmail.com>: Apr 15 07:21AM -0700

This is by design. Consider a replica set with one member in each of 3 data
centers, DC1, DC2, and DC3. Mongo1 is primary and mongo2 and 3 are
secondary. Now a network partition occurs which isolates DC1 but mongo1
stays up. The Mongos in DC2 and DC3 are unaffected, constitute a majority
and vote Mongo2 as primary. If Mongo1 does not step down you will have a
replica set with 2 primaries.... Very bad. Some writes may go to Mongo1
from local users and other writes will go to Mongo2 from users on the other
side of the partition. When the partition clears, it will be very difficult
or impossible to reconcile the two primaries.
The solution is to add data centers and mongo processes until you feel
comfortable that you will always have a majority up :). Or accept that if
you lose two data centers you will have to manually break the replica set
at the surviving data center to allow writes again.
 
On Thursday, April 14, 2016 at 9:32:21 PM UTC-4, Mike Fisher wrote:
Mikko Tommila <mikko....@nitorcreations.com>: Apr 15 03:57AM -0700

My server was running MongoDB 3.2.4 fine but after MongoDB 3.2.5 was
released, now it won't start. The operating system is CentOS 7. When I
start the mongod service, it just immediately fails and there's even
nothing written in the log. If I uninstall the mongodb 3.2.5 packages and
install mongodb-org-3.2.4 then the service starts just fine.
 
The error messages are:
 
[root@ip-192-168-41-156 ~]# service mongod start
Starting mongod (via systemctl): Job for mongod.service failed because the
control process exited with error code. See "systemctl status
mongod.service" and "journalctl -xe" for details.
[FAILED]
 
[root@ip-192-168-41-156 ~]# systemctl status mongod.service
● mongod.service - SYSV: Mongo is a scalable, document-oriented database.
Loaded: loaded (/etc/rc.d/init.d/mongod)
Active: failed (Result: exit-code) since Fri 2016-04-15 10:45:46 UTC;
43s ago
Docs: man:systemd-sysv-generator(8)
Process: 24605 ExecStart=/etc/rc.d/init.d/mongod start (code=exited,
status=1/FAILURE)
Main PID: 13547 (code=exited, status=0/SUCCESS)
 
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Starting SYSV: Mongo is a
scalable, document-oriented database....
Apr 15 10:45:46 ip-192-168-41-156 runuser[24612]:
pam_unix(runuser:session): session opened for user mongod by (uid=0)
Apr 15 10:45:46 ip-192-168-41-156 runuser[24612]:
pam_unix(runuser:session): session closed for user mongod
Apr 15 10:45:46 ip-192-168-41-156 mongod[24605]: Starting mongod: [FAILED]
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: mongod.service: control
process exited, code=exited status=1
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Failed to start SYSV: Mongo
is a scalable, document-oriented database..
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Unit mongod.service entered
failed state.
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: mongod.service failed.
 
[root@ip-192-168-41-156 ~]# journalctl -xe
Apr 15 10:45:14 ip-192-168-41-156 yum[24479]: Installed:
mongodb-org-3.2.5-1.el7.x86_64
Apr 15 10:45:46 ip-192-168-41-156 polkitd[13528]: Registered Authentication
Agent for unix-process:24600:1421059 (system bus name :1.95
[/usr/bin/pkttyagent --n
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Starting SYSV: Mongo is a
scalable, document-oriented database....
-- Subject: Unit mongod.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mongod.service has begun starting up.
Apr 15 10:45:46 ip-192-168-41-156 runuser[24612]:
pam_unix(runuser:session): session opened for user mongod by (uid=0)
Apr 15 10:45:46 ip-192-168-41-156 runuser[24612]:
pam_unix(runuser:session): session closed for user mongod
Apr 15 10:45:46 ip-192-168-41-156 mongod[24605]: Starting mongod: [FAILED]
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: mongod.service: control
process exited, code=exited status=1
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Failed to start SYSV: Mongo
is a scalable, document-oriented database..
-- Subject: Unit mongod.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mongod.service has failed.
--
-- The result is failed.
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Unit mongod.service entered
failed state.
Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: mongod.service failed.
Apr 15 10:45:46 ip-192-168-41-156 polkitd[13528]: Unregistered
Authentication Agent for unix-process:24600:1421059 (system bus name :1.95,
object path /org/free
 
The access rights to the data directory, log directory and tmp directory
seem to be correct.
 
What could be wrong?
 
Mikko
Thomas Burkhart <familiebu...@gmail.com>: Apr 15 02:58AM -0700

Nobody there to help?
"Žygimantas Stauga" <z.st...@gmail.com>: Apr 15 02:41AM -0700

When pricing is not available - usually means "really expensive" :)
 
2016 m. balandis 15 d., penktadienis 11:57:57 UTC+3, Sunghyun Lee rašė:
You received this digest because you're subscribed to updates for this group. You can change your settings on the group membership page.
To unsubscribe from this group and stop receiving emails from it send an email to mongodb-user...@googlegroups.com.

Bernie Hackett

unread,
Apr 15, 2016, 11:46:07 AM4/15/16
to mongodb-user
Since you're using Connection(), you're using PyMongo 2.x. In that case, you want timeout=False, instead of no_cursor_timeout. I would highly recommend upgrading to PyMongo 3.x.

a_l

unread,
Apr 15, 2016, 12:32:38 PM4/15/16
to mongodb-user
Thanks. Should there have been an error on specification of the parameter, it did take me a long while to isolate the issue down to the driver due to the nature of the problem (and because the driver is so consistent otherwise).

I'm using motor, which has an explicit requirement for pymongo to be at 2.8.

Bernie Hackett

unread,
Apr 15, 2016, 2:12:04 PM4/15/16
to mongodb-user
That would be nice. Unfortunately, PyMongo 2.x used **kwargs as a parameter to Cursor to pass some undocumented internal options. Doing validation of unknown options might be ugly, and 2.8 is no longer being developed. You can read the API docs for the 2.8 series here:

https://api.mongodb.org/python/2.8/

PyMongo 2.9.x actually supports no_cursor_timeout. That release series was meant to be a migration path to PyMongo 3.x. At some point Motor will migrate to PyMongo 3.0, but it's not a high priority for us right now.

a_l

unread,
Apr 19, 2016, 7:24:50 AM4/19/16
to mongodb-user
Sounds fair enough, and not a priority.

Would I be correct in thinking that while the motor migration is prioritised, that using gevent in modules that call pymongo in a tornado application would lead to an asynchronous profile roughly similar to that of a motor-driven application?

Bernie Hackett

unread,
Apr 19, 2016, 7:31:44 PM4/19/16
to mongodb-user
Aside from Gevent and Tornado being very different beasts, they will both get you asynchronous IO. Gevent can be used directly with PyMongo (we test this combination in CI, along with Eventlet), whereas Tornado requires Motor.

Reply all
Reply to author
Forward
0 new messages