Account Options

  1. Sign in
Google Groups Home
« Groups Home
Problems with Documents which have high level of nesting
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  13 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Andrew Degtiariov  
View profile  
 More options Feb 12 2010, 5:50 am
From: Andrew Degtiariov <andrew.degtiar...@gmail.com>
Date: Fri, 12 Feb 2010 12:50:15 +0200
Local: Fri, Feb 12 2010 5:50 am
Subject: Problems with Documents which have high level of nesting

There our test case: we defined SessionDocument which contains UserDocument
and UserDocument which contains CompanyDocument in its structure.
Then we creating company, user and session objects and exit.
In another script we trying to load session from DB and get
mongokit.schema_document.SchemaTypeError:

  File "/home/project/test2.py", line 109, in <module>
    s =
database[SessionDocument.collection_name].SessionDocument.find_one({"token" :
u"asddadsad" })
  File "/home/project/mongokit/mongokit/document.py", line 195, in find_one
    return self._obj_class(doc=bson_obj, collection=self.collection)
  File "/home/project//mongokit/mongokit/document.py", line 128, in __init__
    super(Document, self).__init__(doc=doc, gen_skel=gen_skel,
gen_auth_types=False, lang=lang, fallback_lang=fallback_lang)
  File "/home/project/mongokit/mongokit/schema_document.py", line 241, in
__init__
    self._process_custom_type('python', self, self.structure)
  File "/home/project/mongokit/mongokit/schema_document.py", line 562, in
_process_custom_type
    doc[key] = struct[key].to_python(doc[key])
  File "/home/project/mongokit/mongokit/document.py", line 656, in to_python
    return self._doc(doc, collection=col)
  File "/home/project/mongokit/mongokit/document.py", line 137, in __init__
    self._make_reference(self, self.structure)
  File "/home/project/mongokit/mongokit/document.py", line 563, in
_make_reference
    new_path, struct[key]._doc.__name__, type(doc[key]).__name__))
mongokit.schema_document.SchemaTypeError: company must be an instance of
CompanyDocument not DBRef

test1.py:

import mongokit
connection = mongokit.Connection('localhost', 27017)
database = connection['test_case']

class CompanyDocument(mongokit.Document):
    collection_name = "test_companies"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "name": unicode,
    }

    required_fields = [ "name" ]

    indexes = [{
            "fields": "name",
            "unique": True,
            "ttl": 0 # Create index immediately
    }]

class UserDocument(mongokit.Document):
    collection_name = "test_users"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "email": unicode,
        "password": unicode,
        "company": CompanyDocument,
    }

    required_fields = [ "email", "password" ]

    indexes = [
        { "fields": "email",
          "unique": True,

          "ttl": 0 # Create index immediately
        },
    ]

class SessionDocument(mongokit.Document):
    collection_name = "test_sessions"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "token": unicode,
        "owner": UserDocument,
    }

    required_fields = [ "token" ]

    indexes = [
        { "fields": "token",
          "unique": True,
          "ttl": 0 # Create index immediately
        },
    ]

connection.register([CompanyDocument, UserDocument, SessionDocument])
company = database[CompanyDocument.collection_name].CompanyDocument()
company.name = u"Company"
company.save()

company_owner = database[UserDocument.collection_name].UserDocument()
company_owner.email = u"mana...@test.com"
company_owner.password = u"test"
company_owner.company = company
company_owner.save()

s = database[SessionDocument.collection_name].SessionDocument()
s.token = u'asddadsad'
s.owner = company_owner
s.save()

print s

test2.py:
import mongokit
connection = mongokit.Connection('localhost', 27017)
database = connection['test_case']

class CompanyDocument(mongokit.Document):
    collection_name = "test_companies"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "name": unicode,
    }

    required_fields = [ "name" ]

    indexes = [{
            "fields": "name",
            "unique": True,
            "ttl": 0 # Create index immediately
    }]

class UserDocument(mongokit.Document):
    collection_name = "test_users"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "email": unicode,
        "password": unicode,
        "company": CompanyDocument,
    }

    required_fields = [ "email", "password" ]

    indexes = [
        { "fields": "email",
          "unique": True,
          "ttl": 0 # Create index immediately
        },
    ]

class SessionDocument(mongokit.Document):
    collection_name = "test_sessions"
    use_autorefs = True
    use_dot_notation=True

    structure = {
        "token": unicode,
        "owner": UserDocument,
    }

    required_fields = [ "token" ]

    indexes = [
        { "fields": "token",
          "unique": True,
          "ttl": 0 # Create index immediately
        },
    ]

connection.register([CompanyDocument, UserDocument, SessionDocument])

s =
database[SessionDocument.collection_name].SessionDocument.find_one({"token" :
u"asddadsad" })
print s

Execute first test1.py and then execute test2.py

--
Andrew Degtiariov
DA-RIPE


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nicolas Clairon  
View profile  
 More options Feb 12 2010, 8:25 am
From: Nicolas Clairon <clai...@gmail.com>
Date: Fri, 12 Feb 2010 14:25:45 +0100
Local: Fri, Feb 12 2010 8:25 am
Subject: Re: Problems with Documents which have high level of nesting
Hi,

This is indeed a strange behavior. This append when you recreate the
documents and register
them another time. A best practice is to register models only once.
This should fix your issue.
But this is a bug and I'm investigating to get ride of it.

Thanks for the report anyway !

On Fri, Feb 12, 2010 at 11:50 AM, Andrew Degtiariov


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andrew Degtiariov  
View profile  
 More options Feb 12 2010, 8:34 am
From: Andrew Degtiariov <andrew.degtiar...@gmail.com>
Date: Fri, 12 Feb 2010 15:34:13 +0200
Local: Fri, Feb 12 2010 8:34 am
Subject: Re: Problems with Documents which have high level of nesting

2010/2/12 Nicolas Clairon <clai...@gmail.com>

> Hi,

> This is indeed a strange behavior. This append when you recreate the
> documents and register
> them another time. A best practice is to register models only once.
> This should fix your issue.
> But this is a bug and I'm investigating to get ride of it.

> Thanks for the report anyway !

No, I'm registering documents in our test scenario only once. You may look
into scripts test1.py and test2.py
which I included in my first email.
test1.py store documents in DB and test2.py tries to  load one of them.
You should execute test1.py and then test2.py and will see the issue.
--
Andrew Degtiariov
DA-RIPE

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nicolas Clairon  
View profile  
 More options Feb 12 2010, 8:57 am
From: Nicolas Clairon <clai...@gmail.com>
Date: Fri, 12 Feb 2010 14:57:03 +0100
Local: Fri, Feb 12 2010 8:57 am
Subject: Re: Problems with Documents which have high level of nesting
yep, but you redeclared all your classes in the test1.py and test2.py.

If you remove all your classes in test2.py and replace them by `from
test1 import *`,  the test doesn't crash...

I found where is the problem : when you declare and instantiate your
objects for the first time, the `_make_reference` method is called and
transforms all Document with the custom type `R`. When processing the
document with the `_process_custom_type()` method, all `R` objects are
proceed and the conversion DBRef <-> Document is done.

The issue here is while you redeclare all you class, `_make_reference`
is not called (because no Document are created) and Document are not
convert into a R object (so it can not be proceed by
`_process_custom_type`).

I have to found a solution but it is not easy at all. For the moment,
I suggest you to declare you classes only once into a module and make
import in order to use them.

I'll keep you in touch when I'll find a solution.

On Fri, Feb 12, 2010 at 2:34 PM, Andrew Degtiariov


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andrew Degtiariov  
View profile  
 More options Feb 12 2010, 9:08 am
From: Andrew Degtiariov <andrew.degtiar...@gmail.com>
Date: Fri, 12 Feb 2010 16:08:00 +0200
Subject: Re: Problems with Documents which have high level of nesting

2010/2/12 Nicolas Clairon <clai...@gmail.com>

> yep, but you redeclared all your classes in the test1.py and test2.py.

> If you remove all your classes in test2.py and replace them by `from
> test1 import *`,  the test doesn't crash...

Guy, as I say before a scripts should be executed *one after another*. I
mean:
$ python test1.py
$ python test2.py
Traceback (most recent call last):
  File "test2.py", line 109, in <module>
    s =
database[SessionDocument.collection_name].SessionDocument.find_one({"token" :
u"asddadsad" })
  File "/home/project/mongokit/mongokit/document.py", line 195, in find_one
    return self._obj_class(doc=bson_obj, collection=self.collection)
....

Ok, let move a classes into separate module, the is no matter b/c test1.py
is *fully *finished before calling test2.py

PS. test1.py should be called only once, it only filled up collections by
data. Now ANY time when I execute test2.py I got this exception.
Do you think now that the problem in multiple document registration?

I found where is the problem : when you declare and instantiate your

--
Andrew Degtiariov
DA-RIPE

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nicolas Clairon  
View profile  
 More options Feb 12 2010, 9:41 am
From: Nicolas Clairon <clai...@gmail.com>
Date: Fri, 12 Feb 2010 15:41:38 +0100
Local: Fri, Feb 12 2010 9:41 am
Subject: Re: Problems with Documents which have high level of nesting

> Guy, as I say before a scripts should be executed one after another. I mean:
> $ python test1.py
> $ python test2.py

Actually I understood very well what you mean :-)

Anyway, I fixed the bug. I comited it in the last tip (at bitbucket)
so you can give it
a shot and tell me if it works now (it should be).


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andrew Degtiariov  
View profile  
 More options Feb 12 2010, 10:19 am
From: Andrew Degtiariov <andrew.degtiar...@gmail.com>
Date: Fri, 12 Feb 2010 17:19:26 +0200
Local: Fri, Feb 12 2010 10:19 am
Subject: Re: Problems with Documents which have high level of nesting

2010/2/12 Nicolas Clairon <clai...@gmail.com>

> > Guy, as I say before a scripts should be executed one after another. I
> mean:
> > $ python test1.py
> > $ python test2.py

> Actually I understood very well what you mean :-)

> Anyway, I fixed the bug. I comited it in the last tip (at bitbucket)
> so you can give it
> a shot and tell me if it works now (it should be).

No with pymongo 1.4 I got (for test2.py):

Traceback (most recent call last):
  File "test2.py", line 20, in <module>
    connection = mongokit.Connection('localhost', 27017)
  File "/home/project/mongokit/mongokit/connection.py", line 35, in __init__
    super(Connection, self).__init__(*args, **kwargs)
  File
"/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/connection.py",
line 169, in __init__
  File
"/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/connection.py",
line 338, in __find_master
  File
"/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/connection.py",
line 226, in __master
  File
"/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/database.py",
line 220, in command
  File
"/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/collection.py",
line 356, in find_one
  File
"/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/cursor.py", line
485, in next
  File
"/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/cursor.py", line
461, in _refresh
  File
"/opt/project/eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/message.py",
line 110, in query
  File "/opt//eggs/pymongo-1.4-py2.6-linux-i686.egg/pymongo/bson.py", line
572, in from_dict
TypeError: encoder expected a mapping type but got: {'ismaster': 1}

With pymongo 1.3:

Traceback (most recent call last):
  File "test2.py", line 20, in <module>
    connection = mongokit.Connection('localhost', 27017)
  File "/home/project/mongokit/mongokit/connection.py", line 35, in __init__
    super(Connection, self).__init__(*args, **kwargs)
  File
"/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/connection.py",
line 163, in __init__
    self.__find_master()
  File
"/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/connection.py",
line 334, in __find_master
    master = self.__master(sock)
  File
"/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/connection.py",
line 219, in __master
    result = self["admin"]._command({"ismaster": 1}, sock=sock)
  File
"/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/database.py",
line 200, in _command
    _must_use_master=True)
  File
"/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/collection.py",
line 345, in find_one
    _must_use_master=_must_use_master):
  File
"/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/cursor.py", line
480, in next
    if len(self.__data) or self._refresh():
  File
"/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/cursor.py", line
456, in _refresh
    self.__query_spec(), self.__fields))
  File
"/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/message.py",
line 107, in query
    data += bson.BSON.from_dict(query)
  File "/opt/project/eggs/pymongo-1.3-py2.6-linux-i686.egg/pymongo/bson.py",
line 563, in from_dict
    return cls(_dict_to_bson(dict, check_keys))
pymongo.errors.InvalidDocument: documents must have only string keys, key
was 'query'

--
Andrew Degtiariov
DA-RIPE

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nicolas Clairon  
View profile  
 More options Feb 12 2010, 10:37 am
From: Nicolas Clairon <clai...@gmail.com>
Date: Fri, 12 Feb 2010 16:37:51 +0100
Local: Fri, Feb 12 2010 10:37 am
Subject: Re: Problems with Documents which have high level of nesting
I don't understand, it work fine with pymongo 1.4 :

namlook@nichjo:/tmp$ python test1.py
{'owner': {u'company': {u'_id': ObjectId('4b7574f390bce72b4d000000'),
u'name': u'Company'}, u'password': u'test', u'email':
u'mana...@test.com', u'_id': ObjectId('4b7574f490bce72b4d000001')},
'token': u'asddadsad', '_id': ObjectId('4b7574f490bce72b4d000002')}
namlook@nichjo:/tmp$ python test2.py
{u'owner': {u'company': {u'_id': ObjectId('4b7574f390bce72b4d000000'),
u'name': u'Company'}, u'password': u'test', u'email':
u'mana...@test.com', u'_id': ObjectId('4b7574f490bce72b4d000001')},
u'token': u'asddadsad', u'_id': ObjectId('4b7574f490bce72b4d000002')}

Did you pull and update the lasted tip from bitbucket ?

On Fri, Feb 12, 2010 at 4:19 PM, Andrew Degtiariov


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andrew Degtiariov  
View profile  
 More options Feb 12 2010, 10:57 am
From: Andrew Degtiariov <andrew.degtiar...@gmail.com>
Date: Fri, 12 Feb 2010 17:57:22 +0200
Local: Fri, Feb 12 2010 10:57 am
Subject: Re: Problems with Documents which have high level of nesting

2010/2/12 Nicolas Clairon <clai...@gmail.com>

> I don't understand, it work fine with pymongo 1.4 :

Sorry it is work ok when I installed mongokit in cleared environment.
When you planning new release of MongoKit?

--
Andrew Degtiariov
DA-RIPE


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nicolas Clairon  
View profile  
 More options Feb 12 2010, 12:02 pm
From: Nicolas Clairon <clai...@gmail.com>
Date: Fri, 12 Feb 2010 18:02:13 +0100
Local: Fri, Feb 12 2010 12:02 pm
Subject: Re: Problems with Documents which have high level of nesting
Well, a lot of bugs have been fixed so I can release version 0.5.3
tomorrow if you really need to.

On Fri, Feb 12, 2010 at 4:57 PM, Andrew Degtiariov


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andrew Degtiariov  
View profile  
 More options Feb 13 2010, 10:55 am
From: Andrew Degtiariov <andrew.degtiar...@gmail.com>
Date: Sat, 13 Feb 2010 17:55:01 +0200
Local: Sat, Feb 13 2010 10:55 am
Subject: Re: Problems with Documents which have high level of nesting

2010/2/12 Nicolas Clairon <clai...@gmail.com>

> Well, a lot of bugs have been fixed so I can release version 0.5.3
> tomorrow if you really need to.

> Oh, this will great! :-)

--
Andrew Degtiariov
DA-RIPE

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nicolas Clairon  
View profile  
 More options Feb 13 2010, 12:25 pm
From: Nicolas Clairon <clai...@gmail.com>
Date: Sat, 13 Feb 2010 18:25:02 +0100
Local: Sat, Feb 13 2010 12:25 pm
Subject: Re: Problems with Documents which have high level of nesting
Ok, Mongokit v0.5.3 is released.

Enjoy ! :-)

On Sat, Feb 13, 2010 at 4:55 PM, Andrew Degtiariov


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Flaper87  
View profile  
 More options Feb 13 2010, 12:34 pm
From: Flaper87 <flape...@gmail.com>
Date: Sat, 13 Feb 2010 18:34:02 +0100
Local: Sat, Feb 13 2010 12:34 pm
Subject: Re: Problems with Documents which have high level of nesting
Awesome

2010/2/13 Nicolas Clairon <clai...@gmail.com>:

--
Flavio Percoco Premoli, A.K.A. [Flaper87]
http://www.flaper87.org
Usuario Linux registrado #436538
Geek by nature, Linux by choice, Archer of course.
Key Fingerprint: CFC0 C67D FF73 463B 7E55  CF43 25D1 E75B E2DB 15C7
The Solution to everything:
python -c "from struct import pack; print  pack('5b', (41*len('99')),
pow(8,2)+20, 4900**0.5, range(78)[-1], 10)"

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »