New Relic module causing import error

[Yarin]

We're trying to integrate New Relic with web2py. We installed it per their instructions for web2py running on mod_wsgi, adding the following lines in wsgihandler.py:

import newrelic.agent
newrelic.agent.initialize('/opt/web-apps/web2py/newrelic.ini')

right before 

import gluon.main

This seems to work fine, until we import another library our app uses called python-oauth2. Then we get the following error:

    import oauth2 as oauth
  File "/opt/web-apps/web2py/gluon/custom_import.py", line 95, in custom_importer
    raise e2  # there is an error in the module
KeyError: 'httplib2'

I want to report this problem to New Relic, but am unsure what the root cause might be. Is anyone familiar enough with the custom_importer to venture a guess?

[Niphlod]

are you positive that you have httplib2 installed (and available)?

custom_import machinery is tricky and the most complete picture should be in Massimo's end.....

for starters, let's be sure that httplib2 is installed ^_^

[Yarin Kessler]

Yes, we have httplib2 installed in our modules folder, and the python-oauth2 lib runs fine if we don't import the new relic agent first. Likewise, the new relic works fine and we get no errors if we never call import oauth2. It's only when both are imported.

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/5r2YXTa9vAo/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ricardo Pedroso]

On Mon, Apr 29, 2013 at 9:30 PM, Yarin Kessler <ykes...@gmail.com> wrote:
> Yes, we have httplib2 installed in our modules folder, and the python-oauth2
> lib runs fine if we don't import the new relic agent first. Likewise, the
> new relic works fine and we get no errors if we never call import oauth2.
> It's only when both are imported.

What happen if you import httplib2 instead of oauth2 (or before import oauth2)
with newrelic imported first?

Ricardo

> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Yarin]

Ricardo- yes, importing httplib2 throws the same error, even without importing oauth.

We've noticed that the problem does not arise with the new relic import statment on the first line, 

but only on the second line, when the new relic agent is initialized.

import newrelic.agent # Doesn't cause error when httplib2 imported later
newrelic.agent.initialize('/opt/web-apps/web2py/newrelic.ini')# DOES cause error

[Ricardo Pedroso]

On Tue, Apr 30, 2013 at 12:41 AM, Yarin <ykes...@gmail.com> wrote:
> Ricardo- yes, importing httplib2 throws the same error, even without
> importing oauth.
>
> We've noticed that the problem does not arise with the new relic import
> statment on the first line,
> but only on the second line, when the new relic agent is initialized.
>
> import newrelic.agent # Doesn't cause error when httplib2 imported later
> newrelic.agent.initialize('/opt/web-apps/web2py/newrelic.ini')# DOES cause
> error

I really don't know what is happen, just try to help you debug it...

Check if you spot any difference in sys.modules:

import newrelic.agent
import sys
print >>sys.stderr, sys.modules
newrelic.agent.initialize('/opt/web-apps/web2py/newrelic.ini')
print >>sys.stderr, sys.modules


Another option:

What happen when you do:

sys.modules['httplib2'] =
sys.modules['applications.<you_app_name>.modules.httplib2']

before:

import httplib2


Another options can be move httplib2 out of your app.modules


I dont know but It seems that newrelic installs some import hook
and do monkey patching that may be conflicting with web2py custom_import.
taken from http://pydoc.net/Python/newrelic/1.2.0.246/newrelic.config/:

# Register our importer which implements post import hooks for
# triggering of callbacks to monkey patch modules before import
# returns them to caller.

sys.meta_path.insert(0, newrelic.api.import_hook.ImportHookFinder())

Really don't know what is happening, maybe you could ask some support
with newrelic staff.

Ricardo

[Yarin]

Turns out we were able to resolve this simply by pip installing httplib2 instead of loading it from the web2py application directory. Now everything's working together fine. Not sure what the original problem might have been, but thanks for the help--

[Cliff Kachinske]

Are you using a virtualenv?

Maybe you having a version clash between libs.

[Yarin Kessler]

Cliff- No, we are not using virtualenv

On Mon, Apr 29, 2013 at 10:23 PM, Cliff Kachinske <cjk...@gmail.com> wrote:

Are you using a virtualenv?

Maybe you having a version clash between libs.

[Massimo Di Pierro]

Is this only with web2py? What if you import newrelic followed by httplib2 in from shell or in wsgi before importing web2py?

[Yarin Kessler]

As I mentioned earlier this is no longer an issue since we've installed httplib2 via pip. It was happening when httplib2 was being loaded from our app's directory, but I'm honestly not even sure what module version that was. At this point we're just assuming it was a problem with an older module and leaving it at that.