Django 1.5 AbstractBaseUser with char primary key not JSON serializable

[Kaloian]

I am having the following custom user model trying to use the Django 1.5 AbstractBaseUser:

class Merchant(AbstractBaseUser): 
    email = models.EmailField()
    company_name = models.CharField(max_length=256)
    website = models.URLField()
    description = models.TextField(blank=True)
    api_key = models.CharField(blank=True, max_length=256, primary_key=True)   

    USERNAME_FIELD = 'email' 
    REQUIRED_FIELDS = ['company_name','website']


   class Meta:
        verbose_name = _('Merchant')
        verbose_name_plural = _('Merchants')

   def __unicode__(self):
        return self.company_name 

The model works perfectly and database is as expected, but the problem is when I try to dumpdata to create fixtures for my tests. 

python manage.py dumpdata --natural --exclude=contenttypes --exclude=auth.permission --indent=4 > fixtures/initial_data.json

Then I get the error:

CommandError: Unable to serialize database: <Merchant: Test Shop> is not JSON serializable

Do you have ideas what could be the reason for this. Could it be the charfield primary key or something with the abstractbaseuser model?
  
Thanks

[Russell Keith-Magee]

It's not immediately clear. The use of natural keys could also be a contributing factor here. 

What you've described isn't a problem I've seen previously, so you should open a ticket to track it. It would also be exceedingly helpful if you can try running a few tests to remove possible causes - e.g., 

 * Does serialising *without* natural keys work? 

 * Do you have any models with foreign keys to your custom user? (i.e., is the problem manifesting when serialising Merchant, or serialising foreign keys to Merchant?)

 * Do you still have problems if you use a 'normal' integer key?

Essentially, any help you can provide in narrowing down the exact cause would be most helpful.

Also, if you can run the tests with --traceback, we can get the full error logs.

As a side note -- if you're using email as your username field, you should set it as unique=True -- USERNAME_FIELD needs to be unique or you'll experience problems later on. This is something that should probably be caught by validation - which is a bug deserving of it's own report. For performance reasons, you also probably want to set it db_index=True, since you're going to be searching on that field fairly often, so having an index on it makes sense.

Yours,

Russ Magee %-)

[Kaloian]

Hi Russell,


Thanks for the responce! I actually found the problem it was not where I was looking for it and it was more of a typo.
And your suggestion No.2 appears to be right:

* Do you have any models with foreign keys to your custom user? (i.e., is the problem manifesting when serialising Merchant, or serialising foreign keys to Merchant?)

I am having a Product model with foreignkey to the Merchant model which had natural key method defined as follows:

class Product(models.Model):
        name = models.CharField(max_length=200)
        #merchant = models.ForeignKey(Merchant, to_field='api_key')
        merchant = models.ForeignKey(Merchant)
        url = models.URLField(max_length = 2000) 
        description = models.TextField(blank=True)
        client_product_id = models.CharField(max_length='100')

        objects = ProductManager()
        class Meta:
            verbose_name = 'Product'
            verbose_name_plural = 'Products'
            unique_together = ('merchant', 'client_product_id',)

        def __unicode__(self):
            return self.name
 
        def natural_key(self):
            return (self.merchant, self.client_product_id)

The natural_key method returned self.merchant instead of self.merchant_id so it was trying to serialize the whole merchant object to make a natural key.  After switching this to merchant_id it is working properly.

As a side note -- if you're using email as your username field, you should set it as unique=True -- USERNAME_FIELD needs to be unique or you'll experience problems later on. This is something that should probably be caught by validation - which is a bug deserving of it's own report. For performance reasons, you also probably want to set it db_index=True, since you're going to be searching on that field fairly often, so having an index on it makes sense.

Thanks for the notes I will for sure add these, but shouldn't both(unique and db_index) be added by default when you set your USERNAME_FIELD .

Regards,
Kaloian

[Russell Keith-Magee]

On Wed, Feb 13, 2013 at 9:51 PM, Kaloian <kaloian...@gmail.com> wrote:

Hi Russell,


Thanks for the responce! I actually found the problem it was not where I was looking for it and it was more of a typo.
And your suggestion No.2 appears to be right:

* Do you have any models with foreign keys to your custom user? (i.e., is the problem manifesting when serialising Merchant, or serialising foreign keys to Merchant?)

I am having a Product model with foreignkey to the Merchant model which had natural key method defined as follows:

class Product(models.Model):
        name = models.CharField(max_length=200)
        #merchant = models.ForeignKey(Merchant, to_field='api_key')
        merchant = models.ForeignKey(Merchant)
        url = models.URLField(max_length = 2000) 
        description = models.TextField(blank=True)
        client_product_id = models.CharField(max_length='100')

        objects = ProductManager()
        class Meta:
            verbose_name = 'Product'
            verbose_name_plural = 'Products'
            unique_together = ('merchant', 'client_product_id',)

        def __unicode__(self):
            return self.name
 
        def natural_key(self):
            return (self.merchant, self.client_product_id)

The natural_key method returned self.merchant instead of self.merchant_id so it was trying to serialize the whole merchant object to make a natural key.  After switching this to merchant_id it is working properly. 

Glad you found the problem (and even more glad it isn't something I need to fix :-)

 

As a side note -- if you're using email as your username field, you should set it as unique=True -- USERNAME_FIELD needs to be unique or you'll experience problems later on. This is something that should probably be caught by validation - which is a bug deserving of it's own report. For performance reasons, you also probably want to set it db_index=True, since you're going to be searching on that field fairly often, so having an index on it makes sense.

Thanks for the notes I will for sure add these, but shouldn't both(unique and db_index) be added by default when you set your USERNAME_FIELD .

That's an interesting suggestion; I suppose it comes down to a question of "explicit vs implicit". If being named in USERNAME_FIELD magically made the field unique and indexed, the developer needs to implicitly know that this will happen. If a developer actually *wanted* the username field to be non-unique, they might get a surprise when they find that it isn't. Of course, this points at a design problem on their end; the difference is that if it is implicit, they will discover the problem by accident when a non-unique username is rejected; if it is raised as a validation error, they'll get an error when they sync their tables (giving them an indication that the problem must be fixed).

I've opened ticket #19822 to track this problem.

Yours,

Russ Magee %-)

 

[Kaloian]

Hi Russel I got into another small issue related to the character primary key of the user model.
When I try to use the permission_required decorator on that model I got the following error.

Exception Value:	invalid literal for int() with base 10: '3DDz0Vi7zonFDq9JfByKkeparghaWwupVwbMCbL7JHo'
Exception Location:	/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py in get_prep_value, line 554

Here is the latest definition of my model:

class Merchant(AbstractBaseUser,PermissionsMixin): 
    """User with app settings."""
    id = models.CharField(max_length=256, primary_key=True)   
    email = models.EmailField(unique=True, db_index=True)

    company_name = models.CharField(max_length=256)
    website = models.URLField()
    description = models.TextField(blank=True)

    #api_key = models.CharField(blank=True, max_length=256, primary_key=True)  

     
    USERNAME_FIELD = 'email' 
    REQUIRED_FIELDS = ['company_name','website']

    objects = MerchantManager()

    class Meta:
        verbose_name = _('Merchant')
        verbose_name_plural = _('Merchants')

I can easily find workaround for that problem by writing custom decorator for example but I am not sure this is supposed to be this way. Or may be I am missing something here ? 

Thanks,
Kaloian

[Russell Keith-Magee]

Hi Kaloian,

Ah - you've just fired a neuron in my brain… This is a known issue, and it's not just the permission_required decorator.

See ticket #14881.

I don't remember the permission_required decorator itself being a problem at that time (not saying it isn't -- just that it wasn't reported as such, or I don't recall it being reported as such) -- the issues that were being reported were with admin URLs for User objects, and for password reset tokens. 

So - the short answer is that yes, non-integer primary keys will cause some problems with custom User objects. This is something we should probably document as a limitation. I've just added a note to this effect in the docs.

This doesn't mean you have to give up having the api_key as a unique identifier; you just need to have the api_key *in addition* to the integer primary key; Django's internals will use the integer primary key, and your code can do lookups based on the API key.

Of course, we're also open to patches to fix the situation :-)

Yours,

Russ Magee %-)

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kaloian]

Hi Russel,

Great to see this documented. I will most probably go with the "api_key *in addition* to the integer primary key" solution to avoid additional problems.

Thanks,
Kaloian