random import errors

[John-Scott]

I recently migrated 6 sites to a new host (running nginx/apache-mpm-worker/modwsgi 3.3 daemon-mode on Ubuntu 10.4 LTS). They all share the same project directory and virtualenv but each have their own settings file and wsgi config. Only 3 are running Satchmo@fed2d6d6d239, but aside from Satchmo and its dependencies, all other apps are identical across the sites. The old vm was essentially the same, just Ubuntu 9.10 with a Satchmo checkout from October.

The issue is that I'm getting very random errors, "ImportError: cannot import name OrderPayment" (see sanitized stack trace [1]).

This error is extremely random (if I had to guess, 1 in 50-100 requests per site) and I cannot consistently reproduce it (only once have I even triggered one of these errors in all my prodding).

The summary of what's going on in the traceback:

• admin.autodiscover() is in the middle of importing the models from satchmo_store.shops.models.
• While parsing OrderBasePayment, it then needs to lookup a custom field in payment.fields.
• PaymentChoiceField's choices set off a chain of method calls that results in models.get_apps() being called.
• models.get_apps() of course starts the model import process over.
• While models.get_apps() imports payment.models, it attempts to import OrderPayment from satchmo_store.shops, which fails for reasons not clear to me.

Any ideas of what could be causing this random error? Or tips on how to troubleshoot this more effectively?

It seems if this was purely an import error, it would always be an error and not just *sometimes*. At this point it does seem to be some odd combination of Satchmo (or one of its dependencies) and the particulars of my environment since the 3 non-Satchmo sites are working flawlessly. Any keyedcache or threadlocal wonkiness that might be wreaking havoc?

Thanks,
John-Scott

[1] http://dpaste.com/hold/293523/

[Chris Moffitt]

It's interesting you mention this. I noticed something with OrderPayment recently too. I thought it was just the way I was generating the documents but maybe there is something going on here.

I'm wondering if there was a regression introduced recently. One issue might be trying to roll back a ways and see if the error still occurs.

-Chris

--
You received this message because you are subscribed to the Google Groups "Satchmo users" group.
To post to this group, send email to satchm...@googlegroups.com.
To unsubscribe from this group, send email to satchmo-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/satchmo-users?hl=en.

[John-Scott]

Will give this a try. For what it's worth I had been running 2077:1e3e8f3f7795 from 10/18/2010 before upgrading to 2108:fed2d6d6d239 from 12/19/2010.
Given the randomness, will take a while to be convinced it's gone away. Will report back once I've had the chance to make the change and let it bake.

John-Scott

[hynekcer]

On 6 led, 17:32, John-Scott <john.scott.atlak...@gmail.com> wrote:
> While models.get_apps() imports payment.models, it attempts to import
>    OrderPayment from satchmo_store.shops, which fails for reasons not clear to
>    me.

The last line in stack trace tries to import module
satchmo_store.shop.models
which is currently not partially imported because it stops eight
levels in
/home/web/example/env/src/satchmo/satchmo/apps/satchmo_store/shop/
models.py", line 1232, in OrderPaymentBase
and required name OrderPayment is below on that line.
This is recursive dependency.

Something changed the order of module import.
Django usually imports module settings then all INSTALLED_APPS.models
modules and then urls and other things.

All models should be loaded by django before admin.autodiscover()
You can test it and try repair by this code
#
import sys
from django.conf import settings
notloaded =
list(set(settings.INSTALLED_APPS).difference([x.replace('.models','')
for x in sys.modules.keys() if x.endswith('.models')]+
['satchmo_utils.thumbnail']))
# Usually only app satchmo_utils.thumbnail which has no models.py is
not loaded and is excluded from the list.
if notloaded:
print notloaded # something better than print
# you can now try to import listed app.modules or
# if no models are loaded, you can try load them all by some first
usage of database
from django.contrib.sites.models import Site
current_site = Site.objects.get_current()
admin.autodiscover()

What about to try random short delay between new process startup (not
yet full initialised) and the first request? Then probably an issue
for django or mod_wsgi code.

[Jakob H]

Hi,

I get this error on a seemingly regular basis using Django 1.2.4 and
Satchmo tip 0.9.2 (changeset 2115).

Any more information on it?

Cheers,
Jakob

[Chris Moffitt]

I haven't had a chance to look at it in more detail.

-Chris

[John-Scott]

Thanks for your thoughtful and thorough reply, Hynek.

Unfortunately I have been swamped by other projects for the moment and haven't had a spare moment to rigorously troubleshoot (fortunately the errors are rare and as mentioned below do not seem to be triggered by human visitors).

As mentioned in my other thread [1], my issues have mostly vanished. I still occasionally see an import error, although much less frequently. For instance, I had to restart Apache and memcached recently and the first requests to my 3 Satchmo sites each generated a single error email with the import error as the cause. After that, everything was fine. I've also seen more random errors where searchbots hit a completely invalid url on my sites (scratching my head where they've come up with .aspx and .htm urls on a site that has only been Django based and has only had Django-style 'pretty' urls). Don't know that it's cause/effect...sadly the majority of traffic is probably from searchbots to begin with, but the error messages always have googlebot, bingbot or Yahoo! Slurp (gross) as the user agent and the majority of the time they are hitting urls that have never been valid.

Again, haven't hide the time to do another deep dive, but thought I'd toss these anecdotes onto the pile.

Cheers,
John-Scott

[1] https://groups.google.com/d/topic/satchmo-users/4nMkBFw7oGQ/discussion

[Jakob H]

Hi,

I now get this error on basically every request to the website.
However, I'm not sure it is specifically Satchmo related, but I really
need to get this fixed.

The problem does not seem to occur on the runserver on my local
machine, but does happen every time on my shared host deployment
server using WSGI.

Here is my traceback: http://dpaste.com/313255/.

I have Python 2.6.1 on my local machine and Python 2.6.2 on my
deployment server if that matters.

Cheers,
Jakob

On Jan 12, 10:20 pm, John-Scott <john.scott.atlak...@gmail.com> wrote:
> Thanks for your thoughtful and thorough reply, Hynek.
>
> Unfortunately I have been swamped by other projects for the moment and
> haven't had a spare moment to rigorously troubleshoot (fortunately the
> errors are rare and as mentioned below do not seem to be triggered by human
> visitors).
>

> As mentioned in my other thread [1], my issues have* mostly *vanished. I

[Chris Moffitt]

Did you try the solution that was posted earlier in this thread? I haven't tried it yet to see if it fixes it for me.

-Chris

[Jakob H]

Hi,

Not yet, but I will try this tonight. So, the change I will try is:

1. Move class definition of OrderTaxDetail above class definition of
Order
2. Put the FK model reference (Order) within single quotation marks
('Order') to make it load lazily (did not know you could do that)

Does this seem correctly understood?

I will try it and let you know.
Cheers,
Jakob

> > satchmo-user...@googlegroups.com<satchmo-users%2Bunsubscribe@goog­legroups.com>
> > .

[Jakob H]

Hi,

OK, I tried this suggestion but it did not remove the problem. The
same import error still occurs.

Please let me know if you figure out the cause.

Thanks,
Jakob

[Alex Robbins]

Jakob,

Maybe you already know this and are just hoping that someone has
already done the hard work, but this is the plan for fixing this:

If you read the traceback, it looks like cause is a circular import.
Something is importing something that is importing the first thing.
Since both want the other imported first, it blows up. You need to
work through the imports in the traceback, moving them down into
functions so they are delayed, until one of them fixes it.

Hope that helps,
Alex

> To unsubscribe from this group, send email to satchmo-user...@googlegroups.com.

[John-Scott Atlakson]

Hi Jakob,

This rearranging of imports solved a (separate?) issue I was seeing where OrderTaxDetail could not be imported. This had the mysterious effect of eliminating the large number of OrderPayment import errors, even errors I was seeing in sites using an unmodified Satchmo in a separate virtualenv.

I think Chris M was suggesting that you try Hynek's strategy mentioned above.

One simple thing would be to comment out admin.autodiscover() and see if the problem goes away. That doesn't tell us much, but if this is a live site that you need to get back up in a rush, this may be a stop gap.

I may have to take part of this conversation over to the modwsgi group, but there is definitely something specific about Satchmo and/or some of its depencies (keyed-cache, threaded-multihost, etc) since I do not have this problem with any other non-Satchmo projects (even on the same server using the same virtualenv, just no satchmo in INSTALLED_APPS).

Cheers,
John-Scott

To unsubscribe from this group, send email to satchmo-user...@googlegroups.com.

[Jakob H]

Hi,

Thanks for the feedback. I tried Hynek's method, but it didn't help.
The only module that wasn't loaded was 'tax.modules.no' but the
provided code snipped didn't help from what I could see.

What seems to have fixed it for now at least was the following
modification to satchmo/apps/payment/models.py:

1. Comment out "from satchmo_store.shop.models import OrderPayment"
from imports at top of the file (line 15).
2. Do lazy loading of OrderPayment on line 46:

orderpayment = models.ForeignKey('OrderPayment', unique=True,
related_name="creditcards")

Other comments:

1) In my satchmo.log file I see the following:

SettingNotSet: SHIPPING.MODULES
SettingNotSet: PAYMENT_PAYPAL.CREDITCHOICES

Not entirely sure what they refer to or if they are related in any
way.

2) I never see the error when running runserver on my local machine.

3) On my local machine running Apache through WSGI, when I run the
WSGI script (./django.wsgi) I don't get the import error

4) However, on my deployment server (shared hosting) when I run my
WSGI script (./django.wsgi) I get the import error every time (without
my above-mentioned 'fix')

I do not know what the difference is between my local machine (running
Apache) and my shared hosting service that causes the difference in
results. It's the same code, and I've tried to replicate the version
of every library that I know I use and use the same version of
Python.

I'm not really an experienced Python developer so this is all I know
for now. Anyway, it seems to be the cause of a very specific issue of
my deployment configuration, but I don't know what. And, at least the
above-mentioned fix has solved it for me for now.

Again, thanks for your help,
Jakob

On Jan 13, 12:43 pm, John-Scott Atlakson

<john.scott.atlak...@gmail.com> wrote:
> Hi Jakob,
>
> This rearranging of imports solved a (separate?) issue I was seeing where
> OrderTaxDetail could not be imported. This had the mysterious effect of
> eliminating the large number of OrderPayment import errors, even errors I
> was seeing in sites using an unmodified Satchmo in a separate virtualenv.
>
> I think Chris M was suggesting that you try Hynek's strategy mentioned
> above.
>
> One simple thing would be to comment out admin.autodiscover() and see if the
> problem goes away. That doesn't tell us much, but if this is a live site
> that you need to get back up in a rush, this may be a stop gap.
>
> I may have to take part of this conversation over to the modwsgi group, but
> there is definitely something specific about Satchmo and/or some of its
> depencies (keyed-cache, threaded-multihost, etc) since I do not have this
> problem with any other non-Satchmo projects (even on the same server using
> the same virtualenv, just no satchmo in INSTALLED_APPS).
>
> Cheers,
> John-Scott
>

[Chris Moffitt]

Thanks for the suggested fix. Like you, I can only reproduce in unique situations. However, the fix you suggested work for me and didn't break anything else, so I check it in.

The other import errors you're seeing are just debug messages & shouldn't cause any problems.

Thanks again,
Chris

To unsubscribe from this group, send email to satchmo-user...@googlegroups.com.

[John-Scott]

Thanks all. I've reverted my temp modifications and updated the sites to tip, will let you know if this bug rears its head again.

Thanks again,

John-Scott

[Mike S]

I had this exact problem with `OrderPayment`.

It only occurred when importing the `Order` model class in my own
module (which was last in the `INSTALLED_APPS` settings, and only when
running apache+wsgi in my deployment environment.

Your modification of payment/models.py to load `OrderPayment` lazily
fixed the problem.

On Jan 16, 4:24 pm, Jakob H <jakob.henriks...@gmail.com> wrote:
> Hi,
>

> Thanks for the feedback. I tried Hynek's method, but it didn't help.
> The only module that wasn't loaded was 'tax.modules.no' but the
> provided code snipped didn't help from what I could see.
>
> What seems to have fixed it for now at least was the following
> modification to satchmo/apps/payment/models.py:
>
> 1. Comment out "from satchmo_store.shop.modelsimportOrderPayment"
> from imports at top of the file (line 15).
> 2. Do lazy loading of OrderPayment on line 46:
>
> orderpayment = models.ForeignKey('OrderPayment', unique=True,
>         related_name="creditcards")
>
> Other comments:
>
> 1) In my satchmo.log file I see the following:
>
> SettingNotSet: SHIPPING.MODULES
> SettingNotSet: PAYMENT_PAYPAL.CREDITCHOICES
>
> Not entirely sure what they refer to or if they are related in any
> way.
>
> 2) I never see the error when running runserver on my local machine.
>
> 3) On my local machine running Apache through WSGI, when I run the
> WSGI script (./django.wsgi) I don't get theimporterror
>
> 4) However, on my deployment server (shared hosting) when I run my

> WSGI script (./django.wsgi) I get theimporterror every time (without

> > > > > >errorsare rare and as mentioned below do not seem to be triggered by

> > > > > human
> > > > > > visitors).
>
> > > > > > As mentioned in my other thread [1], my issues have* mostly
> > > *vanished. I

> > > > > > still occasionally see animporterror, although much less

> > > frequently.
> > > > > For
> > > > > > instance, I had to restart Apache and memcached recently and the
> > > first
> > > > > > requests to my 3 Satchmo sites each generated a single error email
> > > with
> > > > > the

> > > > > >importerror as the cause. After that, everything was fine. I've also
> > > > > seen
> > > > > > morerandomerrorswhere searchbots hit a completely invalid url on

[hynekcer]

> modification of payment/models.py to load `OrderPayment` lazily
> fixed the problem.

OK. Simple solutions are good for such simple problems.
For the original John's random problem with multithread server is the
randomness the worsest. I think the cause can have many
manifestations. Import errors and all untreated exceptionss are one of
the most desirable :-) easily identifiable manifestations for
debugging. I hope I know a possible cause of problem.

If someone (Chris?) has any random problem with a threaded server and
can describe it so good as John, I will try to prove that the error is
there, where I think. John gave me a very good clue.

Hynek

[John-Scott]

Just wanted to chime in that I have not had a single of these errors since upgrading Satchmo to tip.

Perhaps these import changes just masked over a deeper architectural issue or outright bug. The bottom line for me right now is that the sites work and customers can buy products. I wish I had more time to spare spelunking through the code but just don't have the bandwidth at the moment.

Thanks again to Chris et al for making the changes that got things working again.

Cheers,
John-Scott