Re: loaddata deserializationerror: maximum recursion depth exceeded while calling a Python object

[Ryan Blunden]

Can you provide a single example of one of the fixture objects you're trying to import, as well as the model it corresponds to?

On 29/12/2012, at 11:34 AM, Sam Raker <sam....@gmail.com> wrote:

Hello,

I've got a very tight deadline, and I'm encountering a very frustrating problem. Every time I try to use loaddata to load my data into my database, I get the loaddata error in the subject. I've tried YAML, I've tried JSON, I've tried excerpting only a few lines of each, all to no avail. I'm really at my rope's end. I don't think my models are flawed in any significant way--each refers to only one field in one other model (the primary key in four out of five of them). 

I need to get this working by the 2nd, and I've looked all over the internet and posted a question at Stackoverflow and still can't figure it out. I really need some help.

Thanks,

-sam

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/WLBxL9nsC1IJ.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Sam Raker]

Here's the truncated version I tried to load in case the problem was the length of the file:

- fields: {location: ST. DENNIS HOTEL, restaurant: Veterans American Guard, status: complete,

    year: 1900}

  model: dishes.menu

  pk: 12495

- fields: {location: BOSTON, restaurant: New England Shorthand Reporter's Association,

    status: complete, year: 1900}

  model: dishes.menu

  pk: 12563

- fields: {location: RMS LUCANIA, restaurant: Cunard Line, status: under review, year: 1900}

  model: dishes.menu

  pk: 12749

- fields: {location: IROQUOIS, restaurant: Alumni Association   University Of Buffalo,

    status: complete, year: 1900}

  model: dishes.menu

  pk: 12826

- fields: {location: '[CHICAGO', restaurant: Chicago Bar Association, status: complete,

    year: 1900}

  model: dishes.menu

  pk: 12836

- fields: {location: NEW YORK, restaurant: Hotel Marlborough, status: under review,

    year: 1900}

  model: dishes.menu

  pk: 12841

- fields: {location: NEW YORK, restaurant: Hotel Marlborough, status: complete, year: 1900}

  model: dishes.menu

  pk: 12960

- fields: {location: HOTEL MARLBOROUGH, restaurant: District No.3 Catholic Benevolent

      Legion, status: under review, year: 1900}

  model: dishes.menu

  pk: 13076

- fields: {location: 9 & 10 BATTERY PL. NY, restaurant: Castle Garden Hotel, status: complete,

    year: 1900}

  model: dishes.menu

  pk: 13117

Here's the model:

class Menu(models.Model):

restaurant=models.TextField(unique=False)

year=models.IntegerField(unique=False,null=True)

location=models.TextField(unique=False)

status=models.CharField(unique=False,max_length=20)

pk=models.IntegerField(primary_key=True)

def __unicode__(self):

return restaurant

def __period__(self):#adapted from http://stackoverflow.com/questions/2272149/round-to-5or-other-number-in-python

try:

p=int(10*round(float(self.year)/10))

if p < self.year:

return "%s-%s"%(p,p+5)

else:

return "%s-%s"%(p-5,p)

except:

return ""

period=property(__period__)

language = models.CharField(unique=False,max_length=30)

I'd really appreciate any help on this.

[donarb]

Not sure if it's related to your problem, but you should never create function names like __period__. Names such as those are reserved for Python internals.

[donarb]

And your __unicode__ method is incorrect as well. You should be returning self.restaurant.

[Amirouche]

Some recommandations:

- I never defined __XYZ__ except to look elite ;)

- how «return restaurant» in __unicode__ can possibly work ?

- unique default value = False, no need to repeat it if is not something else

- according to Django codings standards you should not inter-mix model fields with plain properties or methods

- pk=models.IntegerField(primary_key=True) What is it useful for ?

- according to pep8 «except:» should never be used instead use the «except MyException» or «except MyException, e» this prevents bugs [2]

I've got a very tight deadline, and I'm encountering a very frustrating problem. Every time I try to use loaddata to load my data into my database, I get the loaddata error in the subject. I've tried YAML, I've tried JSON, I've tried excerpting only a few lines of each, all to no avail. I'm really at my rope's end. I don't think my models are flawed in any significant way--each refers to only one field in one other model (the primary key in four out of five of them).

How big is the json file ? Which library do you use ? If you use simplejson it's probably that see: https://github.com/simplejson/simplejson/issues/28 This relates to a hierarchical data structure where your datastructure at least the one you pasted is not, I already had troubles with big JSON files and simplejson, I have no metrics sorry.

Can you use SQL ? Maybe you will need to consider spliting the json file into several file.

Regards,

Amirouche

[1] https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style/#model-style

[2] http://peps.io/8/#programming-recommendations

[Sam Raker]

Thanks for your suggestions.

I eliminated the underscores and got rid of the __unicode__ methods entirely, cleaned up my except statement, and changed the order of stuff such that all of each model's fields are defined before any methods. (As for the pk field: The csvs I'm using are actually dumps from another database, so everything has its own primary key already, and I realized it's way easier to just leave them alone rather than waste the processing time and memory reconnecting everything via Python.) 

I'm still having the same problem. I don't think it's related to the length of the files or something in simplejson, as even a dozen-line yaml file gives me the same exception when I try to load it with loaddata. 

Any more ideas?

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/FCFXKNxtvAIJ.

[donarb]

On Sunday, December 30, 2012 9:54:45 AM UTC-8, Sam Raker wrote:

Thanks for your suggestions.

I eliminated the underscores and got rid of the __unicode__ methods entirely, cleaned up my except statement, and changed the order of stuff such that all of each model's fields are defined before any methods. (As for the pk field: The csvs I'm using are actually dumps from another database, so everything has its own primary key already, and I realized it's way easier to just leave them alone rather than waste the processing time and memory reconnecting everything via Python.) 

I'm still having the same problem. I don't think it's related to the length of the files or something in simplejson, as even a dozen-line yaml file gives me the same exception when I try to load it with loaddata. 

Any more ideas?

 

Next, I would post the stack trace, there's probably something in there that points to the problem, if the yaml parser thinks there is recursion in the data. And up the verbosity when running the loaddata command:

--verbosity 3

[Sam Raker]

So I upped the verbosity like you said, and basically all it got me was a bunch of text telling me all the places Django didn't find my fixture before it finally did, and then the same error. Here's the full text of the error:

Traceback (most recent call last):

  File "/home/menusadmin/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 190, in handle

    for obj in objects:

  File "/home/menusadmin/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/site-packages/django/core/serializers/pyyaml.py", line 62, in Deserializer

    raise DeserializationError(e)

DeserializationError: maximum recursion depth exceeded while calling a Python object

I tried changing commit to False in loaddata.py, I tried adding a manager class to the one model I have that another model refers to with a natural key (e.g., 'name,' a CharField, as opposed to the primary key). I read something about loaddata having some unicode-related problems, so I added custom Manager classes for all my models that coerce appropriate fields to strings, e.g.:

class MenuManager(models.Manager):

def create_Menu(self,restaurant,year,location,status,pk,period,language):

menu = self.create(restaurant=str(restaurant),year=int(year),location=str(location),status=str(status),pk=int(pk),period=str(period),language=str(language))

return menu

I'm still getting the exact same error. Help?

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/pq_mI5E031EJ.

[donarb]

On Sunday, December 30, 2012 11:58:46 AM UTC-8, Sam Raker wrote:

So I upped the verbosity like you said, and basically all it got me was a bunch of text telling me all the places Django didn't find my fixture before it finally did, and then the same error. Here's the full text of the error:

Traceback (most recent call last):

  File "/home/menusadmin/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 190, in handle

    for obj in objects:

  File "/home/menusadmin/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/site-packages/django/core/serializers/pyyaml.py", line 62, in Deserializer

    raise DeserializationError(e)

DeserializationError: maximum recursion depth exceeded while calling a Python object

I tried changing commit to False in loaddata.py, I tried adding a manager class to the one model I have that another model refers to with a natural key (e.g., 'name,' a CharField, as opposed to the primary key). I read something about loaddata having some unicode-related problems, so I added custom Manager classes for all my models that coerce appropriate fields to strings, e.g.:

class MenuManager(models.Manager):

def create_Menu(self,restaurant,year,location,status,pk,period,language):

menu = self.create(restaurant=str(restaurant),year=int(year),location=str(location),status=str(status),pk=int(pk),period=str(period),language=str(language))

return menu

I'm still getting the exact same error. Help?

 

Then the next thing I'd do is to test the yml data itself, separate from Django to make sure that the data is not corrupted in any way. Run a script like this, if it passes, then you probably have some sort of error in your models that is recursive.

#!/usr/bin/env python

import yaml

stream = open("test.yml", "r")

print yaml.load(stream)

Finally, I'd run

./manage.py validate

to make sure that all of your models are valid.

 

[Sam Raker]

I just tried both of those things, and the YAML data loaded fine, and validate said I had 0 errors.

Any other suggestions? I'm really stumped here.

 

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/WoMoRX8i3DsJ.

[Sam Raker]

I tried changing my backend to django-mysql-pymysql (http://pypi.python.org/pypi/django-mysql-pymysql/0.1), and that didn't work either. I'm really at my wits' end. Can anyone help?

To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.

[Sam Raker]

I've also tried changing the charset and collation options in my MySQL tables. Still no good. I'm so stumped. Can anyone help me, please?

[donarb]

At this point, I would just skip trying to load the data through Django and write a quick Python script to load the data using INSERT statements or create a bulk data file to load it directly into MySQL. I'm not as familiar with MySQL as I am with Postgres, but Postgres can swallow a whole data file and rollback the transaction should it find invalid data in a row. I'm guessing MySQL can do something similar.