pg8000 driver that comes with web2py 2.8.2 is not supporting JSON field type?

[Horst Horst]

I'm getting a:

  <class 'gluon.contrib.pg8000.errors.NotSupportedError'> type oid 114 not mapped to py type

I'm wondering what's the best thing to do now? Currently I'm considering:

- using TEXT instead. But my former JSON strings are then enclosed by "|" which leads to new errors

- updating gp8000 (there seems to be a newer version, but the version numbering is confusing)

- using psycopg2. How can I install this into an Mac OS standalone version of web2py?

[Mariano Reingart]

You could try to update pg8000 from the official:

https://github.com/mfenniak/pg8000

Let us know if that works, so we could update the one distributed with web2py

The one currently distributed in contrib is an older version with custom patches (as it was not being mantained at the time it was included to web2py), but now the official site has updates that could help you in this case.

Best regards

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Massimo Di Pierro]

Let me know if you think we should upgrade.

[Horst Horst]

I've tried the latest pg8000 as a drop-in replacement, but it seems web2py can't import it:

<type 'exceptions.RuntimeError'> Failure to connect, tried 5 times: Traceback (most recent call last): File "/Users/sfx/dev/mdb/web2py.app/Contents/Resources/gluon/dal.py", line 7766, in __init__ File "/Users/sfx/dev/mdb/web2py.app/Contents/Resources/gluon/dal.py", line 2756, in __init__ File "/Users/sfx/dev/mdb/web2py.app/Contents/Resources/gluon/dal.py", line 795, in find_driver RuntimeError: no driver available ('psycopg2', 'pg8000')

I didn't step through it in the debugger, but the module's __init__.pyc got compiled, so I know web2py attempted an import.

The latest pg8000/__init__.py contains a section which matches the former interface

# For compatibility with 1.8
import pg8000 as dbapi
DBAPI = dbapi
pg8000_dbapi = DBAPI

so I'd guess it's a minor problem, nonetheless I'm lost at this point.

[Mariano Reingart]

Hi Horst:

Sadly there have been a lot of changes in pg8000, so it is not backward compatible with the current custom version in web2py. 

First, you need to delete the pg8000 folder in contrib, and put the new pg8000 folder (the one with __init__.py) directly in the web2py top level folder (at the same level as gluon).

Note that the new pg8000 uses absolute imports that will not work if pg8000 is in contrib folder (or it should be added to the PYTHONPATH)

Second, you need to apply the attached patch to gluon/dal.py to:

 * change the import (and add a missing __version__ attribute)

 * change connection to pass individual parameters (dsn string is not supported anymore)

 * change after_connection set_client_encoding to execute SQL

 * change server_version to _server_version

I'll propose (again) to the pg8000 group the changes I've introduced for web2py, so the latest version could be used as a direct drop-in replacement for psycopg2

IIRC, at some stage the author gave me commit access, but I didn't have time to pull my changes and missed some discussions about the project internals.

Best regards

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

[Joe Barnhart]

Hi Mariano --

I'd like to encourage you to keep making pg8000 a viable alternative for web2py.  I started with it but had to change to psycopg2 because of some issues that postgres handled better under the psy driver.  But I think pypy could be the future of python and it won't support a driver like psycopg2 -- but it could support pg8000 since it is all python.

It's just good to have an alternative because no driver does it all.

-- Joe B.

[Tony Locke]

Hi, I'm a contributor to pg8000 and if there are any changes you need to get pg8000 working with web2py, let me know. Looking at the absolute import problem, PEP8 seems to think absolute imports are the way to go:

http://legacy.python.org/dev/peps/pep-0008/

(search for 'relative' in the text) but I'm open to persuasion :-)

Cheers,

Tony.

[Massimo Di Pierro]

I think you are right. We should include contrib to the sys.path. You should not have to change pg8000 for this.

[Niphlod]

as long as you can use it without installing it in sys.path, then it's fine. If not, it can't be shipped with web2py, just supported as a module.
I don't like packages that needs a complete install to work. It's true that we have venvs for quite some time now, but a module with less than 10 files can accomodate for relative imports quite finely without impairing the code-reader in everyone of us :P

BTW, search for "explicit relative imports" in that doc :-P

[Massimo Di Pierro]

I agree with Niphlod. I do not like packages that need to be installed to work and explicit relative imports is better than absolute imports.

Yet, this is not the only package we had to tweak to include in contrib. So this may come up again. Any objection to include contrib in sys.path?

[Niphlod]

just be sure that the same sys.path is then in scheduler, shell, tests, and so on ^_^
BTW:

from gluon.contrib import something

should be the recommended way to import something from contrib
let's not forget all the hassle we've been into dealing with

from gluon import module

vs

import module

and be smart about it ^___^

BTW2: Isn't including contrib in sys.path just the same thing as shipping pg8000 and all the "nifty importers" in site-packages ?!?!?

[Mariano Reingart]

Including contrib in sys.path would be interesting but there could be collisions (for example, in this case, witch pg8000 will be imported: the one installed or the one in gluon.contrib?)

Tony: there are other changes needed for web2py (see the diff for web2py DAL I've attached earlier), the most important are:

 * missing __version__ attribute (now it is on setup.py, web2py need it to differentiate drivers capabilities)

 * connect doesn't suppor dsn string anymore (you need to pass keyword parameters)

 * set_client_encoding is not present anymore (you need to execute SQL SET ...)

 * server_version attribute has a leading underscore (_server_version), this is need to detect server capabilities like JSON

Also, for the pg8000 driver currently in web2py, I'd applied many bugfixes reported in github / launchpad for the original project (mainly data types, unicode, importing, etc.)

Also, I'd improved the psycopg2 compatibility (i.e. set_client_encoding, autocommit, set_isolation_level), implementing the simple query protocol (the one that uses psycopg2, and to avoid overhead of non-prepared statements) and two-phase commit support.

The last would be important for web2py distributed_transaction_commit, but currently gluon.dal is sending raw SQL PREPARE TRANSACTION / COMMIT PREPARED / ROLLBACK PREPARED without using dbapi proposed methods TPC Connection Methods (tpc_begin, tpc_prepare, tpc_commit, tpc_rollback, tpc_recover) 

You can see the detail of the changes applied to the web2py contrib pg8000 here:

https://code.google.com/p/pg8000/source/list

https://github.com/reingart/pg8000/commits/master

If this kind of changes could be accepted (psycopg2 compatibility, simple query protocol, dbapi 2.0 two-pass commit, etc.), I could help to make the pull requests (note that some could depend on each other). 

Sorry I couldn't keep track of pg8000 due lack of time, so I don't know what direction the project has taken.

Best regards,

Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com

[Massimo Di Pierro]

On Monday, 3 March 2014 14:26:15 UTC-6, Mariano Reingart wrote:

Including contrib in sys.path would be interesting but there could be collisions (for example, in this case, witch pg8000 will be imported: the one installed or the one in gluon.contrib?)

That depends on the order in sys.path. We always put gluon first. gluon.contrib could be second.

Anyway, I agree that unless your other issues are addressed we will stick with your fork.

Massimo

[Tony Locke]

Thanks for your list (and patch) Mariano, that's very useful. I've opened an issue for it at:

https://github.com/mfenniak/pg8000/issues/30

[ArnvShrma]

Hi,

I am facing the exact same issue. Any work-around for this? 

[Niphlod]

until pg8000 supports it, use psycopg2 instead.

[Arnav Sharma]

Hi Niphlod,

I tried using psycopg2 but there are few reasons because of which I can not use it. I am currently deploying travis CI to run Sahana Eden unit tests. psycopg2 fails a few of the tests which it really should not. You can see the build here [1].

[1] https://travis-ci.org/arnavsharma93/eden/jobs/37883503

So, any temporary fix of running pg8000?

You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/BfSIbUSPk38/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Arnav Sharma

IIITH

"If everything is under control, you are going too slow." - Some wise man with big nerdy glasses.

[Niphlod]

On Tuesday, October 14, 2014 11:50:45 PM UTC+2, ArnvShrma wrote:

Hi Niphlod,

I tried using psycopg2 but there are few reasons because of which I can not use it. I am currently deploying travis CI to run Sahana Eden unit tests. psycopg2 fails a few of the tests which it really should not. You can see the build here [1].

[1] https://travis-ci.org/arnavsharma93/eden/jobs/37883503

So, any temporary fix of running pg8000?

web2py does not maintain pg8000 (although its developer is a contributor to web2py's code). Contact him about the issue on the json type.

Also, it seems that most of the errors (web2py doesn't maintain Sahana Eden either, and I'm not a developer or Sahana Eden) comes from lat/long differences from fixed values (again, probably, related from a quick copy/paste unittest suite than a carefully built one)

If only people could READ unittests instead of pointing out failing ones as actual failures....world will have less headaches.
E.g.:

26.0729016786571 != 26.072901678657075

let's confront them carefully... 

26.0729016786571
26.072901678657075

yep. psycopg2 returns the same value than pg8000, only with a lesser precision, estimated in roughly...wait, let's do calcs. 
26.07291 to 26.07292  equals to 1 meter. Every digit you loose in precision, adds up roughly to an order of magnitude... 
meaning 26.0729 to 26.0730 equals 10 meters.
sahana's tests are complaining about a difference of .... well, much more LESS than the dimension a grain of salt (actually, a lot less). 
If sahana is not trying to tie gps coordinates to atoms, psycopg2 is not the problem, sahana's tests are....

the whole concept is even more ridicolous because making tests with FIXED float values is the worst idea ever, especially 
considering that at this level of precision, even your python code can't tell the difference ...

>>> testvalue = 26.072901678657075
>>> print testvalue
26.0729016787
>>> testvalue2 = 26.0729016786570754
>>> testvalue == testvalue2
True

[Anthony]

Is your problem that pg8000 doesn't support the JSON field type, or something else?

[Arnav Sharma]

Hi Niphlod,

I am a newbie to unittests, Eden unittests, postgis, postgresql and its drivers which is why I had come asking for help. Anyhow, I appreciate your input.

Many thanks

On 15 October 2014 03:55, Niphlod <nip...@gmail.com> wrote:

On Tuesday, October 14, 2014 11:50:45 PM UTC+2, ArnvShrma wrote:

Hi Niphlod,

I tried using psycopg2 but there are few reasons because of which I can not use it. I am currently deploying travis CI to run Sahana Eden unit tests. psycopg2 fails a few of the tests which it really should not. You can see the build here [1].

[1] https://travis-ci.org/arnavsharma93/eden/jobs/37883503

So, any temporary fix of running pg8000?

web2py does not maintain pg8000 (although its developer is a contributor to web2py's code). Contact him about the issue on the json type.

Also, it seems that most of the errors (web2py doesn't maintain Sahana Eden either, and I'm not a developer or Sahana Eden) comes from lat/long differences from fixed values (again, probably, related from a quick copy/paste unittest suite than a carefully built one)

If only people could READ unittests instead of pointing out failing ones as actual failures....world will have less headaches.
E.g.:

26.0729016786571 != 26.072901678657075

let's confront them carefully... 

26.0729016786571
26.072901678657075

yep. psycopg2 returns the same value than pg8000, only with a lesser precision, estimated in roughly...wait, let's do calcs. 
26.07291 to 26.07292  equals to 1 meter. Every digit you loose in precision, adds up roughly to an order of magnitude... 
meaning 26.0729 to 26.0730 equals 10 meters.
sahana's tests are complaining about a difference of .... well, much more LESS than the dimension a grain of salt (actually, a lot less). 
If sahana is not trying to tie gps coordinates to atoms, psycopg2 is not the problem, sahana's tests are.

--

Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/BfSIbUSPk38/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paolo Valleri]

Hi,

I created a PR to run test on travis against pg8000 too, https://github.com/web2py/web2py/pull/530. Given that we can check what works and what doesn't.

I'd suggest that you can open an issue here https://github.com/mfenniak/pg8000/issues regarding the missing feature of having json as field type.

Paolo

[Niphlod]

we don't test contrib modules..... pg8000 is the place to make PRs to test, not web2py...

[Anthony]

If the only problem is with JSON support, you might try the following:

db._adapter.types['JSON'] = 'TEXT'

The above is the default when the driver doesn't support JSON (it gets changed to 'JSON' when the DAL thinks the driver supports 'JSON').

If pg8000 doesn't support JSON, then presumably in PostgreSQLAdapter.try_json, the following code is incorrect and should be changed:

        if self.driver_name == "pg8000":
            supports_json = self.connection.server_version >= "9.2.0"

Anthony

To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Niphlod]

PR sent to fix the behaviour with pg8000

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.