Problems with Documents which have high level of nesting

[Andrew Degtiariov]

There our test case: we defined SessionDocument which contains UserDocument and UserDocument which contains CompanyDocument in its structure.
Then we creating company, user and session objects and exit.
In another script we trying to load session from DB and get mongokit.schema_document.SchemaTypeError:

  File "/home/project/test2.py", line 109, in <module>
    s = database[SessionDocument.collection_name].SessionDocument.find_one({"token": u"asddadsad" })
  File "/home/project/mongokit/mongokit/document.py", line 195, in find_one
    return self._obj_class(doc=bson_obj, collection=self.collection)
  File "/home/project//mongokit/mongokit/document.py", line 128, in __init__
    super(Document, self).__init__(doc=doc, gen_skel=gen_skel, gen_auth_types=False, lang=lang, fallback_lang=fallback_lang)
  File "/home/project/mongokit/mongokit/schema_document.py", line 241, in __init__
    self._process_custom_type('python', self, self.structure)
  File "/home/project/mongokit/mongokit/schema_document.py", line 562, in _process_custom_type
    doc[key] = struct[key].to_python(doc[key])
  File "/home/project/mongokit/mongokit/document.py", line 656, in to_python
    return self._doc(doc, collection=col)
  File "/home/project/mongokit/mongokit/document.py", line 137, in __init__
    self._make_reference(self, self.structure)
  File "/home/project/mongokit/mongokit/document.py", line 563, in _make_reference
    new_path, struct[key]._doc.__name__, type(doc[key]).__name__))
mongokit.schema_document.SchemaTypeError: company must be an instance of CompanyDocument not DBRef

test1.py:

import mongokit
connection = mongokit.Connection('localhost', 27017)
database = connection['test_case']

class CompanyDocument(mongokit.Document):
    collection_name = "test_companies"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "name": unicode,
    }

    required_fields = [ "name" ]

    indexes = [{
            "fields": "name",
            "unique": True,
            "ttl": 0 # Create index immediately
    }]

class UserDocument(mongokit.Document):
    collection_name = "test_users"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "email": unicode,
        "password": unicode,
        "company": CompanyDocument,
    }

    required_fields = [ "email", "password" ]

    indexes = [
        { "fields": "email",
          "unique": True,

          "ttl": 0 # Create index immediately
        },
    ]

class SessionDocument(mongokit.Document):
    collection_name = "test_sessions"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "token": unicode,
        "owner": UserDocument,
    }

    required_fields = [ "token" ]

    indexes = [
        { "fields": "token",
          "unique": True,
          "ttl": 0 # Create index immediately
        },
    ]

connection.register([CompanyDocument, UserDocument, SessionDocument])
company = database[CompanyDocument.collection_name].CompanyDocument()
company.name = u"Company"
company.save()

company_owner = database[UserDocument.collection_name].UserDocument()
company_owner.email = u"man...@test.com"
company_owner.password = u"test"
company_owner.company = company
company_owner.save()

s = database[SessionDocument.collection_name].SessionDocument()
s.token = u'asddadsad'
s.owner = company_owner
s.save()

print s

test2.py:
import mongokit
connection = mongokit.Connection('localhost', 27017)
database = connection['test_case']

class CompanyDocument(mongokit.Document):
    collection_name = "test_companies"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "name": unicode,
    }

    required_fields = [ "name" ]

    indexes = [{
            "fields": "name",
            "unique": True,
            "ttl": 0 # Create index immediately
    }]

class UserDocument(mongokit.Document):
    collection_name = "test_users"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "email": unicode,
        "password": unicode,
        "company": CompanyDocument,
    }

    required_fields = [ "email", "password" ]

    indexes = [
        { "fields": "email",
          "unique": True,
          "ttl": 0 # Create index immediately
        },
    ]

class SessionDocument(mongokit.Document):
    collection_name = "test_sessions"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "token": unicode,
        "owner": UserDocument,
    }

    required_fields = [ "token" ]

    indexes = [
        { "fields": "token",
          "unique": True,
          "ttl": 0 # Create index immediately
        },
    ]

connection.register([CompanyDocument, UserDocument, SessionDocument])

s = database[SessionDocument.collection_name].SessionDocument.find_one({"token": u"asddadsad" })
print s

Execute first test1.py and then execute test2.py

--
Andrew Degtiariov
DA-RIPE

[Nicolas Clairon]

Hi,

This is indeed a strange behavior. This append when you recreate the
documents and register
them another time. A best practice is to register models only once.
This should fix your issue.
But this is a bug and I'm investigating to get ride of it.

Thanks for the report anyway !

[Andrew Degtiariov]

2010/2/12 Nicolas Clairon <cla...@gmail.com>

Hi,

This is indeed a strange behavior. This append when you recreate the
documents and register
them another time. A best practice is to register models only once.
This should fix your issue.
But this is a bug and I'm investigating to get ride of it.

Thanks for the report anyway !

No, I'm registering documents in our test scenario only once. You may look into scripts test1.py and test2.py
which I included in my first email.
test1.py store documents in DB and test2.py tries to  load one of them.
You should execute test1.py and then test2.py and will see the issue.

--
Andrew Degtiariov
DA-RIPE

[Nicolas Clairon]

yep, but you redeclared all your classes in the test1.py and test2.py.

If you remove all your classes in test2.py and replace them by `from
test1 import *`, the test doesn't crash...

I found where is the problem : when you declare and instantiate your
objects for the first time, the `_make_reference` method is called and
transforms all Document with the custom type `R`. When processing the
document with the `_process_custom_type()` method, all `R` objects are
proceed and the conversion DBRef <-> Document is done.

The issue here is while you redeclare all you class, `_make_reference`
is not called (because no Document are created) and Document are not
convert into a R object (so it can not be proceed by
`_process_custom_type`).

I have to found a solution but it is not easy at all. For the moment,
I suggest you to declare you classes only once into a module and make
import in order to use them.

I'll keep you in touch when I'll find a solution.

[Andrew Degtiariov]

2010/2/12 Nicolas Clairon <cla...@gmail.com>

yep, but you redeclared all your classes in the test1.py and test2.py.

If you remove all your classes in test2.py and replace them by `from
test1 import *`,  the test doesn't crash...

Guy, as I say before a scripts should be executed one after another. I mean:
$ python test1.py
$ python test2.py
Traceback (most recent call last):
  File "test2.py", line 109, in <module>

    s = database[SessionDocument.collection_name].SessionDocument.find_one({"token": u"asddadsad" })
  File "/home/project/mongokit/mongokit/document.py", line 195, in find_one
    return self._obj_class(doc=bson_obj, collection=self.collection)

....

Ok, let move a classes into separate module, the is no matter b/c test1.py is fully finished before calling test2.py

PS. test1.py should be called only once, it only filled up collections by data. Now ANY time when I execute test2.py I got this exception.
Do you think now that the problem in multiple document registration?

--
Andrew Degtiariov
DA-RIPE

[Nicolas Clairon]

> Guy, as I say before a scripts should be executed one after another. I mean:
> $ python test1.py
> $ python test2.py

Actually I understood very well what you mean :-)

Anyway, I fixed the bug. I comited it in the last tip (at bitbucket)
so you can give it
a shot and tell me if it works now (it should be).

[Andrew Degtiariov]

2010/2/12 Nicolas Clairon <cla...@gmail.com>

> Guy, as I say before a scripts should be executed one after another. I mean:
> $ python test1.py
> $ python test2.py

Actually I understood very well what you mean :-)

Anyway, I fixed the bug. I comited it in the last tip (at bitbucket)
so you can give it
a shot and tell me if it works now (it should be).

No with pymongo 1.4 I got (for test2.py):

Traceback (most recent call last):

  File "test2.py", line 20, in <module>

    connection = mongokit.Connection('localhost', 27017)

  File "/home/project/mongokit/mongokit/connection.py", line 35, in __init__
    super(Connection, self).__init__(*args, **kwargs)
  File "/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/connection.py", line 169, in __init__
  File "/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/connection.py", line 338, in __find_master
  File "/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/connection.py", line 226, in __master
  File "/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/database.py", line 220, in command
  File "/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/collection.py", line 356, in find_one
  File "/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/cursor.py", line 485, in next
  File "/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/cursor.py", line 461, in _refresh
  File "/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/message.py", line 110, in query
  File "/opt//eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/bson.py", line 572, in from_dict
TypeError: encoder expected a mapping type but got: {'ismaster': 1}

With pymongo 1.3:

Traceback (most recent call last):

  File "test2.py", line 20, in <module>

    connection = mongokit.Connection('localhost', 27017)

  File "/home/project/mongokit/mongokit/connection.py", line 35, in __init__
    super(Connection, self).__init__(*args, **kwargs)
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/connection.py", line 163, in __init__
    self.__find_master()
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/connection.py", line 334, in __find_master
    master = self.__master(sock)
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/connection.py", line 219, in __master
    result = self["admin"]._command({"ismaster": 1}, sock=sock)
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/database.py", line 200, in _command
    _must_use_master=True)
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/collection.py", line 345, in find_one
    _must_use_master=_must_use_master):
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/cursor.py", line 480, in next
    if len(self.__data) or self._refresh():
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/cursor.py", line 456, in _refresh
    self.__query_spec(), self.__fields))
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/message.py", line 107, in query
    data += bson.BSON.from_dict(query)
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/bson.py", line 563, in from_dict
    return cls(_dict_to_bson(dict, check_keys))
pymongo.errors.InvalidDocument: documents must have only string keys, key was 'query'

 

--
Andrew Degtiariov
DA-RIPE

[Nicolas Clairon]

I don't understand, it work fine with pymongo 1.4 :

namlook@nichjo:/tmp$ python test1.py
{'owner': {u'company': {u'_id': ObjectId('4b7574f390bce72b4d000000'),
u'name': u'Company'}, u'password': u'test', u'email':
u'man...@test.com', u'_id': ObjectId('4b7574f490bce72b4d000001')},
'token': u'asddadsad', '_id': ObjectId('4b7574f490bce72b4d000002')}
namlook@nichjo:/tmp$ python test2.py
{u'owner': {u'company': {u'_id': ObjectId('4b7574f390bce72b4d000000'),
u'name': u'Company'}, u'password': u'test', u'email':
u'man...@test.com', u'_id': ObjectId('4b7574f490bce72b4d000001')},
u'token': u'asddadsad', u'_id': ObjectId('4b7574f490bce72b4d000002')}

Did you pull and update the lasted tip from bitbucket ?

On Fri, Feb 12, 2010 at 4:19 PM, Andrew Degtiariov

[Andrew Degtiariov]

2010/2/12 Nicolas Clairon <cla...@gmail.com> 

I don't understand, it work fine with pymongo 1.4 :

Sorry it is work ok when I installed mongokit in cleared environment.
When you planning new release of MongoKit?

--
Andrew Degtiariov
DA-RIPE

[Nicolas Clairon]

Well, a lot of bugs have been fixed so I can release version 0.5.3
tomorrow if you really need to.

[Andrew Degtiariov]

2010/2/12 Nicolas Clairon <cla...@gmail.com>

Well, a lot of bugs have been fixed so I can release version 0.5.3
tomorrow if you really need to.

Oh, this will great! :-)

--
Andrew Degtiariov
DA-RIPE

[Nicolas Clairon]

Ok, Mongokit v0.5.3 is released.

Enjoy ! :-)

[Flaper87]

Awesome

2010/2/13 Nicolas Clairon <cla...@gmail.com>:

--
Flavio Percoco Premoli, A.K.A. [Flaper87]
http://www.flaper87.org
Usuario Linux registrado #436538
Geek by nature, Linux by choice, Archer of course.
Key Fingerprint: CFC0 C67D FF73 463B 7E55 CF43 25D1 E75B E2DB 15C7
The Solution to everything:
python -c "from struct import pack; print pack('5b', (41*len('99')),
pow(8,2)+20, 4900**0.5, range(78)[-1], 10)"