IOError when running manage.py update_product_details

[Aakash Desai]

After running manage.py update_product_details (and dj syncdb), I get the following error:

IOError: Unable to load region data for en-US or en-US

The error started popping up after the location patch landed onto master. there something I have to do on local dev to get this running smoothly?

Thanks,
Aakash

[James Socol]

On 6/12/2012 11:34 AM, Aakash Desai wrote:
>
> After running manage.py update_product_details (and dj syncdb), I get
> the following error:
>
> IOError: Unable to load region data for en-US or en-US

First, make sure to update all your submodules. The version of
product_details changed:

git submodule update --init --recursive

Then try running update_product_details with the -f flag. Failing that,
try removing lib/product_details_json/.last_update and running it again.

That should help.

> The error started popping up after the location patch landed onto
> master. there something I have to do on local dev to get this running
> smoothly?

I recommend aliasing gsu='git submodule update --init --recursive'. It's
always a good first step if things seem wrong.

~j

>
> Thanks,
> Aakash
>
>


--
James Socol
Community Platforms Manager
ja...@mozilla.com
@jamessocol

[James Socol]

> That should help.


Did this help?

[Aakash Desai]

Yes and no; I had to get the new product_details files from svn (just like timw). After that, I ran your commands and it all worked sans some db issues (sqlall helped with that).

-- Aakash

[Aakash Desai]

Ah, forgot the most important part: thanks!

[James Socol]

On Wednesday, June 13, 2012 4:46:48 PM UTC-4, Aakash Desai wrote:

Yes and no; I had to get the new product_details files from svn (just like timw). After that, I ran your commands and it all worked sans some db issues (sqlall helped with that).

It interesting that the update_product_details management command was the part that was failing--and apparently failing doing HTTP requests. That makes me suspect some sort of temporary issue with the SVN server or a network issue to it.

[James Socol]

AHA!

One request for the future: please paste full tracebacks. (I'm including this one below.)

I finally ran into this trying to do a git bisect today and figured it out. tl;dr: We've fixed it on master and next. Thanks Owen for the quick r?.

The short version: it's fixed, but not the right way. The long version: there be details/dragons here.

So, in Django, model validation takes place before management commands are allowed to run. Essentially, when Django goes to execute a management command (like update_product_details or runserver), it checks that everything's OK. One of the ways it does this is by importing *.models from everything in INSTALLED_APPS, then checking to see if there's anything totally wrong with those models. It does this by default because it doesn't (necessarily--more on that later) know that the command isn't going to need to use those models, and would rather fail early and loudly.

The UserProfile model--in fact its whole module, because of where this code lives--now depends on a call to product_details.get_regions('en-US') working. Unfortunately, when the regions/en_US.json file isn't there, product details throws a not-very-helpful IOError. This is the IOError everyone was seeing.

So the model validation failed (with the IOError traceback) which then stopped the update_product_details management command from running.

There are a couple ways around this. First, let me explain what y'all did. Then I'll explain the quick hack I did to make the builds work. Then I'll explain the real fix we'll do.

What y'all did
You all cloned the product_details data directly off SVN. That meant that regions/en_US.json was there and so model validation passed, so the command could run. And everything was hunky-dory. That's fine, but it's a pain for new contributors, and it's a solution that would be really hard to document for other product-details users.

The Quick and the Dirty
I checked in lib/product_details_json/regions/en_US.json. You can force things past .gitignore, and that's what I did here. Essentially, it lets the model validation pass, so the management command can run. It works. I don't like it, but it works. And it worked today to get stage deployed. (Again, thanks Owen!)

Slightly Better--in the Future
Some recent commit to Django gives management commands a --skip_validation option by default. If we were on a new-enough Django, that would've been a good work-around. But it's not ideal because it involves a lot of typing. Every time.

What we Should and Will Do
Django anticipated this. Management commands that don't touch models (like update_product_details) have a mechanism for skipping validation already (every time, it's not like the option I just described). If you set requires_model_validation = False on your BaseCommand subclass, it skips the global model validation. Booyah.

So I'm opening a pull request with django-mozilla-product-details right now to add that attribute and then no one should have to deal with this again. And we'll revert the commit I made once the pull req is merged and we've updated DMPD in our vendor repo again.

And for posterity, here's the full traceback:

Traceback (most recent call last):
  File "manage.py", line 24, in <module>
    manage.main()
  File "/home/james/mozillians/vendor/src/funfactory/funfactory/manage.py", line 121, in main
    execute_manager(current_settings)
  File "/home/james/mozillians/vendor/src/django/django/core/management/__init__.py", line 442, in execute_manager
    utility.execute()
  File "/home/james/mozillians/vendor/src/django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/james/mozillians/vendor/src/django/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/james/mozillians/vendor/src/django/django/core/management/base.py", line 219, in execute
    self.validate()
  File "/home/james/mozillians/vendor/src/django/django/core/management/base.py", line 249, in validate
    num_errors = get_validation_errors(s, app)
  File "/home/james/mozillians/vendor/src/django/django/core/management/validation.py", line 31, in get_validation_errors
    for (app_name, error) in get_app_errors().items():
  File "/home/james/mozillians/vendor/src/django/django/db/models/loading.py", line 158, in get_app_errors
    self._populate()
  File "/home/james/mozillians/vendor/src/django/django/db/models/loading.py", line 65, in _populate
    self.load_app(app_name, True)
  File "/home/james/mozillians/vendor/src/django/django/db/models/loading.py", line 89, in load_app
    models = import_module('.models', app_name)
  File "/home/james/mozillians/vendor/src/django/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/james/mozillians/apps/users/models.py", line 31, in <module>
    class UserProfile(SearchMixin, models.Model):
  File "/home/james/mozillians/apps/users/models.py", line 57, in UserProfile
    choices=product_details.get_regions('en-US').items(),
  File "/home/james/mozillians/vendor/src/django-mozilla-product-details/product_details/__init__.py", line 101, in get_regions
    raise IOError('Unable to load region data for %s or en-US' % locale)

[James Socol]

A brief update on this, which is now resolved correctly (or will be, as soon as the pull request gets merged).

 

The Quick and the Dirty
I checked in lib/product_details_json/regions/en_US.json. You can force things past .gitignore, and that's what I did here. Essentially, it lets the model validation pass, so the management command can run. It works. I don't like it, but it works. And it worked today to get stage deployed. (Again, thanks Owen!)

I've undone this. lib/product_details_json now ships as an empty directory with just a .gitignore file in it. As nature intended.

 

What we Should and Will Do
Django anticipated this. Management commands that don't touch models (like update_product_details) have a mechanism for skipping validation already (every time, it's not like the option I just described). If you set requires_model_validation = False on your BaseCommand subclass, it skips the global model validation. Booyah.

This pull request merged, and we updated product-details in our vendor repo. Now, update_product_details should work out of the box, and the error that model validation throws will be a little more useful.