Can not import copy_reg

[Richard]

Hi,

I have an issue with adjusting the auth tables resulting in a custom import and an error that copy_reg can not be imported.

The error trace is:

Traceback (most recent call last):
  File "/home4/mamplcom/public_html/cgi-bin/gluon/restricted.py", line 220, in restricted
    exec ccode in environment
  File "/home4/mamplcom/public_html/cgi-bin/applications/im/models/db.py", line 74, in <module>
    auth.define_tables(username=False, signature=False)
  File "/home4/mamplcom/public_html/cgi-bin/gluon/tools.py", line 1784, in define_tables
    format='%(first_name)s %(last_name)s (%(id)s)'))
  File "/home4/mamplcom/public_html/cgi-bin/gluon/dal.py", line 8226, in define_table
    table = self.lazy_define_table(tablename,*fields,**args)
  File "/home4/mamplcom/public_html/cgi-bin/gluon/dal.py", line 8263, in lazy_define_table
    polymodel=polymodel)
  File "/home4/mamplcom/public_html/cgi-bin/gluon/dal.py", line 1125, in create_table
    sql_fields_old = pickle.load(tfile)
  File "/home4/mamplcom/public_html/cgi-bin/gluon/custom_import.py", line 92, in custom_importer
    return base_importer(pname, globals, locals, fromlist, level)
ImportError: No module named copy_reg

I can import copy_reg from the python prompt and adding the import copy reg from Dal.py (suggested in a thread by Massimo) has no effect.

I use Python 2.7.8.

Any clu would be helpfull.

Richard

[Massimo Di Pierro]

Copy reg is a python module. Can you import it from the normal python shell?

[Richard]

Yes, I can import copy_reg from the Python prompt on the remote server.

[Massimo Di Pierro]

Can you try from the web2py shell?

python web2py.py -S welcome -P

[Richard]

When I add the import copy_reg in db.py I get the error:

<type 'exceptions.ImportError'> No module named copy_reg

In the python/Python-2.7.8/Lib directory are 3 copy_reg files:

- copy_reg.py

- copy_reg.pyc

- copy_reg.pyo

Maybe of any relevance: I run a shared Apache server with the following first line in cgihandler.py :  #!/home4/mamplcom/python/Python-2.7.8/python

On Tuesday, July 22, 2014 1:21:36 PM UTC+2, Massimo Di Pierro wrote:

[Massimo Di Pierro]

I understand but can you open a web2py shell. from the console type:

$ cd web2py

$ python web2py.py -S welcome -P

>>> import copy_reg

[Richard]

Massimo,

importing copy_reg from the console works fine :)

Richard

[Niphlod]

I'd vote for the usual issue with apache and mod_wsgi not compiled against the latest interpreter....

[Richard]

The issue does not occur on my local development machine. 

I did an upgrade on the remote host where the issue occurs using the web2py update feature. Then I upgraded from Python 2.6 to 2.7.8. All apps works fine but not the one where I add features to auth.

How should resolve this in a manner that I have a solid basis for production? We are in a pre-production phase.

[Niphlod]

I seem to be the only "advocate" for ditching apache always and before anything else (especially before going into production, where a not-so-well-tuned apache (most-likely you're using everybody's else defaults) still suffers a Slowloris attack).
Those kind of headaches are non-existent with other solutions (often more performant without particular tuning).
Check with a simple google search for "mod_wsgi recompile" to see how many people have been in your situation (mismatch between what mod_wsgi has been originally compiled vs the current running python version) and their solutions for various distributions. It's a pretty well documented "issue" with a large number of cases.

[Richard]

Thank you for the answer, I did hope for an other answer but i'll go in the Google mode.

i'll keep you informed about the progress.

[Massimo Di Pierro]

I agree with you that Apache is not a good solution any more. Nginx for example is a much better solution. Nevertheless some people may not have a choice.

[Richard]

I am hosted at JustHost, who has the option on installing Nginx. I'll try that for a better infrastructure.

[Richard]

nginx will need a VPS so i'll stick to CGI for prototyping and buy some time for a new hosting service. 

In the book an option for "Some web hosting services do not support mod_wsgi. In this case, we must use Apache as a proxy and forward all incoming requests to the web2py built-in web server (running for example on localhost:8000)."

I'll try this first.

[Richard]

While setting up a new server I am wondering why previous versions of my app worked fine.

What functionality triggers the custom_import trying to import copy_reg? A temporarily solution might miss this functionality and I'll build this back when the hosting is in place again.

[Richard]

Some progress is not what I expect :( . It takes time to learn how to configure a webserver.

I am now hosting the app at Pythonanyware, I expect the libraries they use are up to date. assuming they are, also at pythonanyware running python2.7 the same error occurs.

Could it be a bug in Web2py? Please assist me in diagnosing and testing so nobody else will run in this issue.

Richard

[Giles Thomas]

Hi all,

Giles from PythonAnywhere here.  I've been working with Richard to try to track this down further.  One thing I can confidently say is that it's not an Apache/mod_wsgi problem in this case, because we use nginx and uWSGI.

I've also double-checked that the `copy_reg` import works from a trivial Flask app, and that was fine, so I don't think it's the interpreter.

Does anyone have any thoughts on how we could further debug this?

All the best,

Giles

[Leonel Câmara]

My guess would be this is happening because of you copying files with pickles around. Some stuff is not portable, for instance, web2py uses marshal in some places.

Pickle is infamous for giving very weird error messages like the one you are getting that appear to point to other problems.

If I am right, a solution can be to delete everything with pickles on it. Like the table files in the database folder, the sessions, and everything in the cache.

[Giles Thomas]

Thanks, Leonel.  We'll look into that.

On 30 July 2014 18:12, Leonel Câmara <leonel...@gmail.com> wrote:

My guess would be this is happening because of you copying files with pickles around. Some stuff is not portable, for instance, web2py uses marshal in some places.

Pickle is infamous for giving very weird error messages like the one you are getting that appear to point to other problems.

If I am right, a solution can be to delete everything with pickles on it. Like the table files in the database folder, the sessions and everything in the cache.

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/8toVDprfqwM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Giles Thomas

gi...@giles.net

http://www.gilesthomas.com/

http://www.pythonanywhere.com/

[Richard]

Progress made :) On PythonAnywhere I can clean cache, sessions and database and start storing data again. But packing the app importing on my development machine, editing and packing and restoring on PA gives the same error. Some thing on my machine is wrong I'll find out! :)

Happily it is no bug in Web2py.

Leonel, Giles, thank you for the support !

Richard 

[Leonel Câmara]

Richard if packing and unpacking the application in pythonanywhere is causing problems the cause may be because .pyc files are going with your application pack (you can open the application pack with any archive application and check), maybe in your modules folder. If that's the case this could probably be considered a web2py bug.

[Niphlod]

well, you can't expect a compiled app to work amongst python versions..... you should compile it on the destination ....

[Leonel Câmara]

I agree niphlod, what I am saying may be a bug is that admin should probably make sure pyc files do not go anywhere in the application pack by default.

[Niphlod]

uhm.....I don't think anybody put a triple check for it.... but you may want to ship a compiled app without the sourcecode available for your customers, e.g. to update a fixed release.

[Leonel Câmara]

Frankly, I think it's borderline retarded to distribute python applications without the source code. Firstly, pyc files are not portable, secondly pyc files are trivial to reverse using something like https://github.com/Mysterie/uncompyle2, and thirdly, your app may actually stop working if your client upgrades Python unless you're also bundling your own Python interpreter.  

  

Basically, if you don't want people to have access to your source code, you shouldn't be using Python, or, if you really don't want to give the source to the clients, you should be forcing your clients to use your own hosting that you control.

  

For these reasons, I don't think distributing as pycs should even be an option.

[Richard]

My current situation is that I can move the software with no sqlite database to PA, fill the database, looks ok, not efficient though.

Packing the app with no errors, no chache, but sqlite file, gives the import error.

Zip instead of pack has the same result.

But distribution using zip/unzip without the database gives no error. In fact this will be the case in most cases in my current situation.

Maybe something is wring in adding extra fields to auth. I do the following:

auth.settings.extra_fields['auth_user']= [ 

    Field('community', 'reference community', IS_IN_DB(db,'community.id', '%(com_name)s (%(id)s)'), label=T('Community')),

    Field('langu', 'reference langu', requires=IS_IN_DB(db,'langu.id','%(langu)s'))]

auth.settings.extra_fields['auth_group']= [

    Field('community', 'reference community', label=T('Community'),

    requires=IS_EMPTY_OR(IS_IN_DB(db,'community.id','%(com_name)s (%(id)s)'))) ]

auth.define_tables(username=False, signature=False)

Where this last line for the auth table creation raises the can not import copy_reg error.

[Leonel Câmara]

Richard basically it's the .table files in your database folder that are creating the problem. You can set migrate=False in your DAL call on db.py  before packing the application to prevent the problem, then turn it back on in your development machine. 

  

Table files is where web2py stores your table's structure to detect changes and do migrations, since the structure of the database is already going to be correct as any migration has already been done in your development machine there's no need to do it in the server and have these problems.

[Richard]

Leonel, switching migrate to False works indeed perfect in my case. Thank you :)
Richard