Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carsten Fuchs]

Dear Django fellows,

using Django 1.8.3, in a fully migrated app I have a model with a many-to-many
relationship like this:

from django.db import models
from django.contrib.auth.models import User

class Bereich(models.Model):
benutzer = models.ManyToManyField(User)

Now I would like to store some extra information with the relationship (e.g. some
Booleans that describe what the user is permitted to do with the related Bereich object).
That is, the intended result is planned to look like this:

class Bereich(models.Model):
benutzer = models.ManyToManyField(User, through='UserBereichAssignment')

class UserBereichAssignment(models.Model):
bereich = models.ForeignKey(Bereich)
user = models.ForeignKey(User)
can_edit = models.BooleanField(verbose_name="can edit?", default=True)

However:

$ ./manage.py makemigration # ok, creates a new migrations file
$ ./manage.py migrate

# ...
ValueError: Cannot alter field Lori.Bereich.benutzer into Lori.Bereich.benutzer -
they are not compatible types (you cannot alter to or from M2M fields, or add or remove
through= on M2M fields)


Well, I understand that and also seem to see some of the involved problems: For example,
how would it make sure that the same database table is used for the previous implicit
intermediate model and the new UserBereichAssignment?

I'm also aware of https://code.djangoproject.com/ticket/23034, but I'm not sure what to
make of it, or if it is even applicable here.


Thus, what would be a good way to proceed?

To the best of my understanding, I could do this manually, that is: create an entirely
new many-to-many field, e.g.

class Bereich(models.Model):
benutzer = models.ManyToManyField(User) # old, unchanged
benutzer_ = models.ManyToManyField(User, through='UserBereichAssignment')

then manually migrate the data from "benutzer" to "benutzer_" (possibly using a data
migration?), and finally remove the old field "benutzer". The only downside seems that
all dependent code had to be updated from using the old name "benutzer" to the new name
"benutzer_". (I'm not sure if making a migration for renaming the new field name to the
old field name is possible?)

Alas... are there any viable alternatives to this?
I'd be very grateful for any hint! :-)

Best regards,
Carsten

[Gergely Polonkai]

You don't need the temporary field.

1) create the "through" model, add it to the migration file
2) migrate all m2m instances (e.g. iterate over all Bereich objects then iterate through its users, creating a UserBereichAssignment object for each (all this done in a migrations.RunPython call)
3) change the m2m field by adding the through option, and add this change to the migrations file.

However, you may want to go with django-guardian and thus, with object-level permissions instead ;)

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/55A7BA3E.5030105%40cafu.de.
For more options, visit https://groups.google.com/d/optout.

[Carsten Fuchs]

Hi Gergely,

many thanks for your quick reply!

Am 2015-07-16 um 18:15 schrieb Gergely Polonkai:
> 1) create the "through" model, add it to the migration file
> 2) migrate all m2m instances (e.g. iterate over all Bereich objects then
> iterate through its users, creating a UserBereichAssignment object for
> each (all this done in a migrations.RunPython call)

Ok. I've never used migrations.RunPython before, but I'm quite confident
that I can manage it.

Would the migration for step 2) be a separate migration file from step
1), or is the migration file of step 1) edited to cover step 2) as well?

> 3) change the m2m field by adding the through option, and add this
> change to the migrations file.

Same question: Is this supposed to go into a separate migration file or
into the common migrations file from above?

And won't this last step trigger the same error as before? ("... you

cannot alter to or from M2M fields, or add or remove through= on M2M

fields") ?

(Not so important, but out of curiosity: This won't get us rid of the
old, implicit intermediate table, will it?)

> However, you may want to go with django-guardian and thus, with
> object-level permissions instead ;)

Thanks for mentioning this!

I've started looking into it, but the only user-code example that I've
been able to find so far is in the middle of
https://github.com/djangoadvent/djangoadvent-articles/blob/master/1.2/06_object-permissions.rst
Are there any other examples that demonstrate the purpose and usage of
this package?


Many thanks and best regards,
Carsten

[BHAGYARAJ POLA]

hey hi,

I want to create web based visualization by using django framework in python.

 i need to have a front end interface which will communicate with the database.

The front end should look like similar to this.

http://specobs.ee.washington.edu/

For Map you can use Google Maps API, See how to use that and you know most of the information.

From my view:

Google Maps should take the location data from your input

Input will be latitude, longitude and channel number

Output will be the heat map with the PSD value from the database.

Can u help me in this task?

thanx,

[Carsten Fuchs]

Hi Gergely,

Am 16.07.2015 um 20:44 schrieb Carsten Fuchs:
>> 3) change the m2m field by adding the through option, and add this
>> change to the migrations file.
>

> [...]

> And won't this last step trigger the same error as before? ("... you cannot alter to or
> from M2M fields, or add or remove through= on M2M fields") ?

Would you mind clarifying and elaborating on your reply please?

Most importantly, why would step 3) not result in the same error as I originally
experienced?

[Carsten Fuchs]

Am 16.07.2015 um 16:05 schrieb Carsten Fuchs:
> Alas... are there any viable alternatives to this?
> I'd be very grateful for any hint! :-)

Anyone please?

Best regards,
Carsten

[Carl Meyer]

Hi Carsten,

On Thursday, July 16, 2015 at 12:44:55 PM UTC-6, Carsten Fuchs wrote:

Am 2015-07-16 um 18:15 schrieb Gergely Polonkai:
> 1) create the "through" model, add it to the migration file
> 2) migrate all m2m instances (e.g. iterate over all Bereich objects then
> iterate through its users, creating a UserBereichAssignment object for
> each (all this done in a migrations.RunPython call)

Ok. I've never used migrations.RunPython before, but I'm quite confident
that I can manage it.

You could also use RunSQL instead of RunPython for this data migration step - personally I often find that easier (and almost always more efficient) for a simple data migration, since all you need is a SQL query to copy some data from one table to another, you don't need any Python conveniences or model instances along the way.

But either can work, it really just depends how comfortable you are with raw SQL.
 

Would the migration for step 2) be a separate migration file from step
1), or is the migration file of step 1) edited to cover step 2) as well?

Separate migration. In general you can't put schema alterations and data migrations in the same migration file, because then they'll try to run in the same transaction, and PostgreSQL at least doesn't like that.

In general, for complex changes, this three-step dance (one migration to add the new field/table, a second migration to copy the data, and a third migration to remove the old field/table) is very common and the right way to do things.
 

> 3) change the m2m field by adding the through option, and add this
> change to the migrations file.

Same question: Is this supposed to go into a separate migration file or
into the common migrations file from above?

And won't this last step trigger the same error as before? ("... you
cannot alter to or from M2M fields, or add or remove through= on M2M
fields") ?

This part I'm not sure about without trying it. I'm honestly not sure what exactly Gergely was recommending. Based on my reading of the code, the error is raised by the schema-editor backend, meaning if you try an `AlterField` with this change, you'd hit the error.

Another possibility might be to use the `SeparateDatabaseAndState` operation to create a migration that has no effect on the schema, but just updates the Python state for the field. Since you've already made the necessary db schema changes simply by adding the through model itself, this should work fine.

You could also go back and edit the field definition in whichever migration initially created it. This would probably work, but it would cause problems for any `RunPython` migrations since then that used that field (because now they'd try to use the through table instead), including your own data migration that you just created prior to this. So probably not a good option.
 

(Not so important, but out of curiosity: This won't get us rid of the
old, implicit intermediate table, will it?)

No. You could add another RunSQL migration to remove this table using raw SQL.

Overall I think it might be simpler to go with your initial approach. I'd first rename the old field to some other temporary name (you don't have to update any other code to use this name, as you'll commit all of these migrations in one commit, so nothing but the following data migration ever needs to know anything about this temporary name), then add the new field with through table (using the original field name), then migrate data (using either RunPython or RunSQL), then remove the old field. I think this will work fine, and it should also remove the old auto-created through table.

Good luck! I think this is definitely an area where the migrations system could help a bit more.

Carl

[Carsten Fuchs]

Hi Carl,

Am 23.07.2015 um 18:28 schrieb Carl Meyer:
> Overall I think it might be simpler to go with your initial approach. I'd first rename
> the old field to some other temporary name (you don't have to update any other code to
> use this name, as you'll commit all of these migrations in one commit, so nothing but
> the following data migration ever needs to know anything about this temporary name),

Thanks for clarifying the details regarding renaming! With this, this is the approach
that I feel the most comfortable with and will try now.

Overall, many thanks for your thoughts and your detailed reply, it's very much
appreciated and helps a lot!

Best regards,
Carsten

[Gergely Polonkai]

Hello,

I'm sorry, I was not around my mailbox lately.

Yes, you are right, my attempt is not the solution to your problem; it seems that this m2m field really cannot be modified like this. With some slight modifications, though, it may be.

1) create the through table
2) migrate data with RunPython — if you want to be portable, stay with RunPython instead of RunSQL
3) delete the ManyToManyField
4) recreate the ManyToManyField with the through option

All this can (and I think, should) go into the same migration file. Meanwhile it might worth a question to the devs (or a more thorough search on the topic) to understand why you can't switch from a simple m2m field to one with a through option.

Best,
Gergely

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/55B11F7B.1010805%40cafu.de.

[Carl Meyer]

Hi Gergely,

On 07/23/2015 02:27 PM, Gergely Polonkai wrote:
> Yes, you are right, my attempt is not the solution to your problem; it
> seems that this m2m field really cannot be modified like this. With some
> slight modifications, though, it may be.
>
> 1) create the through table
> 2) migrate data with RunPython — if you want to be portable, stay with
> RunPython instead of RunSQL

Yes, this is true. Don't use RunSQL if you want your code to be portable
between database backends. (For most of my non-reusable-apps code, I
take great advantage of Postgres-specific features and don't care at all
about cross-database portability.)

> 3) delete the ManyToManyField
> 4) recreate the ManyToManyField with the through option
>
> All this can (and I think, should) go into the same migration file.

Hmm. I'm almost sure I've had problems in the past with trying to do
schema alterations and data migration in the same migration file (thus
same transaction), but with a simple test just now it seems to work OK
on Postgres. I'll have to start trying to combine migrations like this
more often and see how it goes.

> Meanwhile it might worth a question to the devs (or a more thorough
> search on the topic) to understand why you can't switch from a simple
> m2m field to one with a through option.

Just because it's a bit complex to implement, and nobody has implemented
it yet.

Carl

[Carsten Fuchs]

Hi all,

just wanted to let you know that I was successful!

I followed Gergely's final recipe, using RunPython for the data migrations and separate
migration files for each of the steps.

For step 3 (the deletion of the ManyToManyField) I temporarily had to remove the
django.contrib.admin from the INSTALLED_APPS, because I was referring to the old
ManyToManyField in my app's admin.py setup, which in turn was processed while running
`./manage.py makemigrations`.

Overall, this worked very well.

Many thanks to you both for your help!
:-)

Am 23.07.2015 um 22:37 schrieb Carl Meyer:
>> Meanwhile it might worth a question to the devs (or a more thorough
>> search on the topic) to understand why you can't switch from a simple
>> m2m field to one with a through option.
>
> Just because it's a bit complex to implement, and nobody has implemented
> it yet.

Well, if I understand this correctly, https://code.djangoproject.com/ticket/23034 has a
reviewed PR that implements this, but unfortunately the original author has not updated
the PR for several months...

Best regards,
Carsten