MongoDB Adapter error in select

[Francisco Costa]

So I have this:

import sys
import time
from gluon.dal import DAL, Field
mongo = DAL('mongodb://localhost:27017/tymr')
mongo.define_table('user',
             Field('name', 'text'),
             Field('age',  'integer'),
             Field('city', 'string')
             )

def insert_users():
    mongo.user.insert(name='John', age=66, city='Toronto')
    mongo.user.insert(name='Mark', age=43, city='Boston')
    mongo.user.insert(name='Tom',  age=43, city='Detroit')
    mongo.user.insert(name='Jim',  age=18, city='Detroit')
    mongo.user.insert(name='Jack', age=18)
    mongo.user.insert(name='Eric', city='Boston')
    return 'users in database'

def find_users():
    users = mongo(mongo.user.age==66).select()
    return dict(users=users)

after I run insert_users I check in MongoDb via terminal and everything is correct:

> db.user.find()
{ "_id" : ObjectId("4fa80feea34feb34f8000000"), "city" : "Toronto", "age" : NumberLong(66), "name" : "John" }
{ "_id" : ObjectId("4fa80feea34feb34f8000001"), "city" : "Boston", "age" : NumberLong(43), "name" : "Mark" }
{ "_id" : ObjectId("4fa80feea34feb34f8000002"), "city" : "Detroit", "age" : NumberLong(43), "name" : "Tom" }
{ "_id" : ObjectId("4fa80feea34feb34f8000003"), "city" : "Detroit", "age" : NumberLong(18), "name" : "Jim" }
{ "_id" : ObjectId("4fa80feea34feb34f8000004"), "age" : NumberLong(18), "name" : "Jack" }
{ "_id" : ObjectId("4fa80feea34feb34f8000005"), "city" : "Boston", "name" : "Eric" }

but when I run find_users I get this error:

  File "/opt/web2py/gluon/dal.py", line 7578, in select
    return adapter.select(self.query,fields,attributes)
  File "/opt/web2py/gluon/dal.py", line 4290, in select
    return self.parse(rows,fields,mongofields_dict.keys(),False)
  File "/opt/web2py/gluon/dal.py", line 1600, in parse
    self.parse_value(value, fields[j].type,blob_decode)
  File "/opt/web2py/gluon/dal.py", line 1496, in parse_value
    return self.parsemap[key](value,field_type)
  File "/opt/web2py/gluon/dal.py", line 1562, in parse_id
    return int(value)
ValueError: invalid literal for int() with base 10: 'Toronto'

[Massimo Di Pierro]

Your linennumbers do not much the code in trunk. Please try the code in trunk. Anyway, something goes wrong in the parse(self, rows, fields, colnames, blob_decode=True) function. I suspect mongo does not return the columns in the order web2py think it does. Can you print the input of the parse function?

[Francisco Costa]

I think that the mongofields_dict.keys() don't match to the data in the rows

mongofields_dict	:	id name age city
rows	:	Toronto 66L 24652490551171733682233802752L John

[Massimo Di Pierro]

Thanks for your help and check on this. I just committed a fix in trunk that should address it. Please let me know.

[Francisco Costa]

missing a colon at the end of the line 4677

if key == 'id'

and now I get this error

  File "/opt/web2py/gluon/dal.py", line 8134, in select
    return adapter.select(self.query,fields,attributes)
  File "/opt/web2py/gluon/dal.py", line 4711, in select
    return processor(rows,columns,False)
  File "/opt/web2py/gluon/dal.py", line 1675, in parse
    for j,colname in enumerate(colnames):
TypeError: 'bool' object is not iterable


maybe because of the False on line 4711

[Massimo Di Pierro]

Please try again.

[Francisco Costa]

Now i get this error:

  File "/opt/web2py/gluon/dal.py", line 8134, in select
    return adapter.select(self.query,fields,attributes)
  File "/opt/web2py/gluon/dal.py", line 4711, in select

    return processor(rows,fields,colnames,False)
  File "/opt/web2py/gluon/dal.py", line 1681, in parse
    self.parse_value(value, fields[j].type,blob_decode)
  File "/opt/web2py/gluon/dal.py", line 1575, in parse_value
    return self.parsemap[key](value,field_type)
  File "/opt/web2py/gluon/dal.py", line 1641, in parse_id

    return int(value)
ValueError: invalid literal for int() with base 10: 'Toronto'

Return in this order:

colnames	:	city age _id name
fields	:	<gluon.dal.Field object at 0x24fe0d0> <gluon.dal.Field object at 0x1b25790> <gluon.dal.Field object at 0x186cbd0> <gluon.dal.Field object at 0x24fe090>

rows

:

Toronto 66L 24652490551171733682233802752L John

[Francisco Costa]

I've enumerated fields and colnames and they are like this:

fields

:

id text integer string

colnames

:

city age _id name

 I believe they don't match and thats why there's this error:

File "/opt/web2py/gluon/dal.py", line 8134, in select
return adapter.select(self.query,fields,attributes)
File "/opt/web2py/gluon/dal.py", line 4711, in select

return processor(rows,fields,colnames,False)
File "/opt/web2py/gluon/dal.py", line 1681, in parse
self.parse_value(value, fields[j].type,blob_decode)
File "/opt/web2py/gluon/dal.py", line 1575, in parse_value
return self.parsemap[key](value,field

_type)
  File "/opt/web2py/gluon/dal.py", line 1641, in parse_id

[kokoyo]

Hi  Costa , first im sorry for my wrong post above,It should be a new post with new question. Actually i have same problem like you, 

i did a quick test and have not solved this issue yet, here is what i did:

set driver to use mongodb in: DB.py :

if not request.env.web2py_runtime_gae:

    ## if NOT running on Google App Engine use SQLite or other DB

    #db = DAL('sqlite://storage.sqlite')

    db = DAL('mongodb://localhost:27017/foo') 

Click Register on menu to put a new User into mongo, submit:

    <type 'exceptions.ValueError'> invalid literal for int() with base 10: 'koko'

Go to web2py appadmin admindatabase, open auth_user table:

Invalid Query 

invalid literal for int() with base 10: 'koko' 

Use mongo console to run query :

db.auth_user.find({})

new user is alread inserted into db,--> "insert" is done.

but the error happens when mapping value and key or key type in DAL, im not sure, the type of value is not match. looks like DAL understands ID field is FirstName field.

• File C:\w2py3\gluon\dal.py in parse_id at line 1641 code arguments variables

  Function argument list

  (self=<gluon.dal.MongoDBAdapter object>, value='koko', field_type='id')

  Code listing

  1636.
  1637.
  1638.
  1639.
  1640.
  1641.
  
  1642.
  1643.
  1644.
  1645.
          if not self.dbengine=='google:datastore':
              value = bar_decode_string(value)
          return value
  
      def parse_id(self, value, field_type):
          return int(value)
  
  
      def parse_integer(self, value, field_type):
          return int(value)
  

  Variables

  builtinint	<type 'int'>
  value	'koko'

[Francisco Costa]

Lets hope Massimo fix that today in trunk

[khuc viet Cuong]

just re-edit old post (typo+bad english), Seems DAL is alway looks the first "column" of all tables is ID field   (remove _id column from mongo)

        1                                 2                              3

{  "first_name" : "koko", "last_name":"yo", "password"........ }

       1                                                                               2                                3                        4

{ "_id" : ObjectId("4de83c816afb62e8367163b1"), "first_name" : "koko", "last_name":"yo", "password"........ }

[Massimo Di Pierro]

Should be an easy fix but I need your help. Before line 4711 of dal.py

return processor(rows,fields,colnames,False)

can you add

print rows

print fields

print colnames

what is the output when it fails?

[kokoyo]

Hi Massimo, i did it and get output (test on user_auth table):

mongoqry_dict ={'email': 'mye...@gmail.com'}

mongofields_dict = SON([('id':1), ('first_name':1),  ('last_name':1),  ('email':1), ('password':1), ('registration_key':1), ('reset_password_key':1), ('registration_id':1) ])

mongo_list_dicts = <pymongo.cursor.Cursor object at 9a809aa>

[Francisco Costa]

I've enumerated fields and colnames and they are like this:

fields

:

id text integer string

colnames

:

city age _id name

 I believe they don't match and thats why there's this error:

File "/opt/web2py/gluon/dal.py", line 8134, in select
return adapter.select(self.query,fields,attributes)
File "/opt/web2py/gluon/dal.py", line 4711, in select

return processor(rows,fields,colnames,False)
File "/opt/web2py/gluon/dal.py", line 1681, in parse
self.parse_value(value, fields[j].type,blob_decode)
File "/opt/web2py/gluon/dal.py", line 1575, in parse_value
return self.parsemap[key](value,field

_type)
  File "/opt/web2py/gluon/dal.py", line 1641, in parse_id



    return int(value)
ValueError: invalid literal for int() with base 10: 'Toronto'

Francisco Costa
http://franciscocosta.com

[Massimo Di Pierro]

Please try trunk again.

[Francisco Costa]

Still not working. I think Fields order is wrong.

At the end of the select() function I write this:

        a = []
        for f in fields:
            a.append(f.type)
        return dict(rows=rows, fields=a, colnames=colnames)

and I got this:

colnames	:	city age _id name
fields	:	id text integer string

[Massimo Di Pierro]

This helps. I have another attempt to fix this in trunk. Hard to test it since I do not have mongodb installed.

[Francisco Costa]

Just add a '\' at the end of line 4664 and your are done!

Great job Massimo!

[Francisco Costa]

I found another problem!!

if you do this:

    mongo.user.insert(name='Jim',  age=18, city='Detroit')
    mongo.user.insert(name='Jack', age=18)

and then select
    users = mongo(mongo.user.age==18).select()

you don't have the field "city" in one of the rows, so you get this error:

    users = mongo(mongo.user.age==18).select()
  File "/opt/web2py/gluon/dal.py", line 8123, in select
    return adapter.select(self.query,fields,attributes)
  File "/opt/web2py/gluon/dal.py", line 4700, in select

    return processor(rows,fields,colnames,False)
  File "/opt/web2py/gluon/dal.py", line 1681, in parse
    self.parse_value(value, fields[j].type,blob_decode)
  File "/opt/web2py/gluon/dal.py", line 1575, in parse_value

    return self.parsemap[key](value,field_type)
  

File "/opt/web2py/gluon/dal.py", line 1641, in parse_id
    return int(value)

ValueError: invalid literal for int() with base 10: 'Jack'

[Massimo Di Pierro]

What's the model?

[Francisco Costa]

i didn't understand your question..
This is the complete code

import sys
import time
from gluon.dal import DAL, Field

mongo = DAL('mongodb://localhost:27017/sapo')

mongo.define_table('user',
                     Field('name', 'text'),
                     Field('age',  'integer'),
                     Field('city', 'string')
                     )

def insert_users():
    mongo.user.insert(name='John', age=66, city='Toronto')
    mongo.user.insert(name='Mark', age=43, city='Boston')
    mongo.user.insert(name='Tom',  age=43, city='Detroit')

    mongo.user.insert(name='Jim',  age=18, city='Detroit')
    mongo.user.insert(name='Jack', age=18)

    mongo.user.insert(name='Eric', city='Boston')
    return 'users in database'

def find_users():

    users = mongo(mongo.user.age==18).select()
    return dict(users=users)

[Massimo Di Pierro]

Now I understand better where all these problems come from. I shall fix it tonight.

[Francisco Costa]

Thanks Massimo. Please let me know if you need more help in debugging it

[Francisco Costa]

the latest DAL produces this error

    users = mongo(mongo.user.age==18).select()
  File "/opt/web2py/gluon/dal.py", line 8004, in select
    return adapter.select(self.query,fields,attributes)
  File "/opt/web2py/gluon/dal.py", line 4528, in select
    colnames = [fix(column) for column in mongo_list_dicts[0]]
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 402, in __getitem__
    raise IndexError("no such item for Cursor instance")
IndexError: no such item for Cursor instance

[Massimo Di Pierro]

One more try please.

[Francisco Costa]

It works Massimo!
In MongoDb is also common to insert dicts, can we also have that?

At the moment it only saves in the string format (check player Field)
{ "_id" : ObjectId("4fae3839b34f4brf5a000031"), "city" : "Madrid", "club" : "Real Madrid", "player" : "{'n_goals': 43, 'name': 'Ronaldo'}" }

[Massimo Di Pierro]

I think you can already

db.table.insert(**dict(....))

Are you asking for something different?

[Krzysztof Mulica]

I'm guessing he's looking for something like I am, 

http://cookbook.mongodb.org/patterns/count_tags/ 

{
    "title" : "A blog post",
    "author" : "Kristina",
    "content" : "...",
    "tags" : ["MongoDB", "Map/Reduce", "Recipe"]
}

using DAL, how would I go about inserting that list, what wouldd the type be in the model definition?

[Francisco Costa]

requested here: https://groups.google.com/d/topic/web2py/pqdYpUbMXKQ/discussion