move database to mongodb

[黄祥]

it seems move existing code that use a rdms (sqlite, mysql, postgresql) to mongodb need an extra effort.

e.g.

DAL connection from book

>>> db = DAL('mongodb://user:password@localhost/connect_test')

Error:

...

OperationFailure: command SON([('authenticate', 1), ('user', u'user'), ('nonce', u'7f46671524655a7d'), ('key', u'2be426ffba7c067a8a4ddb2a5e5056bf')]) on namespace connect_test.$cmd failed: auth failed

Solution:

1. without mongodb user and password

db = DAL('mongodb://localhost/connect_test')

or 

2. create mongodb user and password in terminal shell

cat <<EOL > userMongoDB.js

use connect_test

db.createUser(

  {

    user: "username",

    pwd: "password",

    roles: [ { role: "dbAdmin", db: "connect_test" },

             { role: "userAdmin", db: "connect_test" } ] 

  }

)

EOL

mongo test < userMongoDB.js

Date not str

>>> db.table0.insert(date0 = "2017-01-01", datetime0 = "2017-01-01 00:00:00")

Error:

...

TypeError: combine() argument 1 must be datetime.date, not str

Solution (strange that datetime field type didn't get affected by an error traceback):

from datetime import datetime

db.table0.insert(date0 = datetime.strptime('2017-01-01', '%Y-%m-%d'), datetime0 = "2017-01-01 00:00:00")

Iterselect

>>> rows = db().iterselect(db.table0.ALL)

Error:

...

AttributeError: 'Mongo' object has no attribute 'iterselect'

Solution:

rows = db().select(db.table0.ALL)

not sure is there any difficulties i'll face during the moving, but grateful a lot things to learn from obstacles.

best regards,

stifan

[黄祥]

Can't retrieve value of reference field (tested in views and in models using represent)

{{=DIV('%s - %s' % (row_person.id_card.id_card_type, row_person.id_card.id_card_number) ) }}

Error:

...

RuntimeError: Using a recursive select but encountered a broken reference: id_card 1

Solution:
change row_person.id_card.id_card_type into row_person.id_card (just show the id not the value)

Cant' run execute_sql() to create index

db.executesql('CREATE INDEX idx_auth_user ON auth_user (id, first_name, last_name, email, username);')

Error:
...

AttributeError: 'Mongo' object has no attribute 'execute'

Solution: 

commented it for now, check on pydal github, there is a solution for this, but don't know how to create an mongodb index in pydal, any hints? 

best regards,

stifan

[黄祥]

Code:

auth.add_permission(auth_user_id_1, "impersonate", "auth_user", auth_user_id_2)

Error:

...

OverflowError: MongoDB can only handle up to 8-byte ints

there was a ticket 2 years ago, but still occured, tested in web2py 2.15.4 

ref:

https://github.com/web2py/pydal/issues/223

best regards,

stifan

[Richard Vézina]

Hello Steve,

I had read about MongoDB for a project that I had in mind... When I see your thread first question come to mind is why porting relational database to mongo... In mongo you will have to design your database in a much different way then in and relational engine in order to make some benefit from it. As far as I understand you end in having much less tables as it the key to improve the speed of the back end...

So I am not sure why you try to simply migrate an actual database design into mongo...

Richard

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[黄祥]

just want to learn about mongodb and how to use it with web2py, the same code is work in all rdms (tested in sqlite, mysql, postgre except change configuration db uri in appconf.ini and the executesql() create index for text field in mysql (minor) ) but when tested with nosql (first try with mongodb) a lot of things must be changed in my existing code.

yes you are right, i think that the application is a bit slower when using mongodb rather than using rdms, perhaps the reason is what you said 'having much less tables as it the key to improve the speed of the back end'

n.b.

when an error occured during testing with existing code, just tested in new app the suspected error code to make it simple and can focus what is going wrong and how to fix it

thanks and best regards,

stifan

[Dave S]

My understanding of MongoDB is that it is supposed to be fast to WRITE to, making it good for recording things that happen in a stream.

I've only used it in a node.js/React context, and only for one assignment, so I'm not able to provide you with much detail.

/dps
 

[Richard Vézina]

I listen a few vid tutorial about data modeling it mongoDB and basically you should think to put in a single document (mongo is a document database important to remember) all (or most of them) the piece of data into a single document base on the use case of these data (application view for instance)... It not fully clear to my understanding if there is any structured way to determine how to include data or not in a given entity (document), I didn't apply my learning yet so... But I guess you should balance between the size and the amount of data you need to show base on the view of your application... Even the way you create your app maybe influenced by the fact that speed of the backend is not suppose be the bottleneck anymore or it limit is much higher...

I think mongo is a good nosql choice but I doubt that migrating relational model into mongo will bring you any benefit without remodeling your data structure... You should also consider using nosql type of database for particula piece of data but before on board and other database engine in your stack you have to consider it as it's not a trivial choice. For this reason and because Postgres offer nosql type of data structure (hstore) you may or may not be better to stick with it depends of your needs.

Good luck

Richard

--

[黄祥]

thank you all for the input, this topic is open just to put what i tested when using mongodb (just started to learn nosql trying first with mongodb that is supported by web2py), perhaps there are some issue is not resolved like auth permission

another error testing found :

1. Didn't support multiple table selected (for report purpose)

Code :

return test_report.report_0(query_0, query_1, query_2)

Error Traceback:

...

RuntimeError: Too many tables selected (['table0', 'table1', 'table2'])

2. Didn't parse data correctly to highcharts (no error occured, but the graphic shown is not correct)

the same code with the same data input, works well in sqlite (for all tested above), i think the first test can be ignored since richard already explain it to put it into a single document.

best regards,

stifan

[Richard Vézina]

I think the issue you talk about is related or is the same one to the one report here : https://groups.google.com/forum/#!topic/web2py/oPcn_TFEJvQ

Richard

--

[Dave S]

On Tuesday, September 26, 2017 at 7:15:08 AM UTC-7, 黄祥 wrote:

thank you all for the input, this topic is open just to put what i tested when using mongodb (just started to learn nosql trying first with mongodb that is supported by web2py), perhaps there are some issue is not resolved like auth permission

If I recall correctly, there's a GAE no-sql db that's supported in web2py, but that's about all I can tell you about it.  For AWS, I've only used RDBs, so I'm not sure what's in play there for no-sql.  I don't think Elastic Search is considered a DB as much as an analyzer.

[One of the other benefits of MongoDB is they have a cloud version in their own-branded cumulus. Saves you having to set up Yet Another Server.]

/dps