[Django] #24853: django.db.utils.IntegrityError: null value in column "obj_class_id" violates not-null constraint
Django
violates not-null constraint
----------------------------------------------+----------------------------
Reporter: rakanalh | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer (models, ORM) | Version: 1.7
Severity: Normal | Keywords: database,
| migrations
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+----------------------------
Hello,
I am using "django_dynamic_scraper" to scrape data off the internet. I
started my project by creating an empty migration that basically adds a
default scraper.
Models as defined in dynamic_scraper app:
{{{
class ScrapedObjClass(models.Model):
name = models.CharField(max_length=200)
scraper_scheduler_conf = models.TextField(default='\
"MIN_TIME": 15,\n\
"MAX_TIME": 10080,\n\
"INITIAL_NEXT_ACTION_FACTOR": 10,\n\
"ZERO_ACTIONS_FACTOR_CHANGE": 20,\n\
"FACTOR_CHANGE_FACTOR": 1.3,\n')
checker_scheduler_conf = models.TextField(default='\
"MIN_TIME": 1440,\n\
"MAX_TIME": 10080,\n\
"INITIAL_NEXT_ACTION_FACTOR": 1,\n\
"ZERO_ACTIONS_FACTOR_CHANGE": 5,\n\
"FACTOR_CHANGE_FACTOR": 1.3,\n')
comments = models.TextField(blank=True)
def __unicode__(self):
return self.name
class Meta:
ordering = ['name',]
class ScrapedObjAttr(models.Model):
ATTR_TYPE_CHOICES = (
('S', 'STANDARD'),
('T', 'STANDARD (UPDATE)'),
('B', 'BASE'),
('U', 'DETAIL_PAGE_URL'),
('I', 'IMAGE'),
)
name = models.CharField(max_length=200)
obj_class = models.ForeignKey(ScrapedObjClass)
attr_type = models.CharField(max_length=1, choices=ATTR_TYPE_CHOICES)
def __unicode__(self):
return self.name + " (" + self.obj_class.__unicode__() + ")"
}}}
Here's my migration:
{{{
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models, migrations
def add_youtube_scraper(apps, schema_editor):
Scraper = apps.get_model('dynamic_scraper', 'Scraper')
ScrapedObjClass = apps.get_model('dynamic_scraper', 'ScrapedObjClass')
ScrapedObjAttr = apps.get_model('dynamic_scraper', 'ScrapedObjAttr')
ScraperElem = apps.get_model('dynamic_scraper', 'ScraperElem')
scraped_obj_class = ScrapedObjClass()
scraped_obj_class.name = 'Youtube Video'
scraped_obj_class.save()
scraped_attrs_map = {}
scraped_attrs_list = [
{'name': 'base', 'type': 'B'},
{'name': 'url', 'type': 'U'},
{'name': 'title', 'type': 'S'},
{'name': 'body', 'type': 'S'},
{'name': 'images', 'type': 'I'},
{'name': 'videos', 'type': 'S'},
]
for scraped_attr in scraped_attrs_list:
scraped_obj_attr = ScrapedObjAttr()
scraped_obj_attr.name = scraped_attr['name']
scraped_obj_attr.attr_type = scraped_attr['type']
scraped_obj_attr.obj_class_id = scraped_obj_class.id
scraped_obj_attr.save()
scraped_attrs_map[scraped_attr['name']] = scraped_obj_attr
class Migration(migrations.Migration):
dependencies = [
]
operations = [
migrations.RunPython(add_youtube_scraper)
]
}}}
When i run python manage.py migrate i get the following error:
{{{
Running migrations:
Applying content_scraper.0001_initial...
Traceback (most recent call last):
File "./manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/site-
packages/django/core/management/__init__.py", line 385, in
execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/site-
packages/django/core/management/__init__.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/site-
packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/site-
packages/django/core/management/base.py", line 338, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/site-
packages/django/core/management/commands/migrate.py", line 161, in handle
executor.migrate(targets, plan, fake=options.get("fake", False))
File "/usr/local/lib/python2.7/site-
packages/django/db/migrations/executor.py", line 68, in migrate
self.apply_migration(migration, fake=fake)
File "/usr/local/lib/python2.7/site-
packages/django/db/migrations/executor.py", line 102, in apply_migration
migration.apply(project_state, schema_editor)
File "/usr/local/lib/python2.7/site-
packages/django/db/migrations/migration.py", line 108, in apply
operation.database_forwards(self.app_label, schema_editor,
project_state, new_state)
File "/usr/local/lib/python2.7/site-
packages/django/db/migrations/operations/special.py", line 117, in
database_forwards
self.code(from_state.render(), schema_editor)
File "/app/content_scraper/migrations/0001_initial.py", line 32, in
add_youtube_scraper
scraped_obj_attr.save()
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py",
line 589, in save
force_update=force_update, update_fields=update_fields)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py",
line 617, in save_base
updated = self._save_table(raw, cls, force_insert, force_update,
using, update_fields)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py",
line 698, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk,
raw)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py",
line 731, in _do_insert
using=using, raw=raw)
File "/usr/local/lib/python2.7/site-
packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py",
line 921, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 921, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-
packages/django/db/backends/utils.py", line 82, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/usr/local/lib/python2.7/site-
packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line
94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-
packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "obj_class_id"
violates not-null constraint
DETAIL: Failing row contains (19, base, null, B).
}}}
Basically what i am trying to do is assign a value to a ForeignKey. I
tried printing the value of the created scraper object class and it's 19
as in the last line of the traceback. I also checked the SQL statement
generated:
{{{
INSERT INTO "dynamic_scraper_scrapedobjattr" ("name", "attr_type") VALUES
(%s, %s) RETURNING "dynamic_scraper_scrapedobjattr"."id"
}}}
As you can see the obj_class attribute of the ScrapedObjAttr is omitted,
which is a weird behaviour.
--
Ticket URL: <https://code.djangoproject.com/ticket/24853>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: rakanalh | Owner: nobody
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by shaib):
* needs_better_patch: => 0
* needs_docs: => 0
* needs_tests: => 0
Comment:
Hello,
Thanks for taking the trouble to submit this problem report. Some points
arise:
1) Per Django's maintenance policy, only major bugs (security issues or
data-loss problems) will be fixed in 1.7. Can you please try to reproduce
the issue with the current 1.8 release?
2) While you're at it, if the problem still exists, and to help pinpoint
it, please try to replace the line
{{{
scraped_obj_attr.obj_class_id = scraped_obj_class.id
}}}
with
{{{
scraped_obj_attr.obj_class = scraped_obj_class
}}}
In principle, it should be equivalent.
3) Could you please clarify how you dealt with django-dynamic-scraper's
migrations? The current source has a migrations folder, but those are
South migrations, incompatible with Django>=1.7.
Thanks,
Shai.
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:1>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Is the migration with `RunPython` the initial migration, i.e.
`0001_initial.py`? If so, this is incorrect -- you need an initial
migration which creates all the models in the app.
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:2>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by shaib):
Replying to [comment:2 timgraham]:
> Is the migration with `RunPython` the initial migration, i.e.
`0001_initial.py`? If so, this is incorrect -- you need an initial
migration which creates all the models in the app.
If I understand correctly, the migration belongs to `content_scraper` (an
app with no models of its own?) but it uses the models from
`dynamic_scraper`, which appears to be an unmigrated app (its public
repository has only South migrations). This should be valid, as far as I
understand, but only if my assumptions indeed hold.
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by rakanalh):
Thanks for the follow up on this ticket.
I started off the project using Django 1.6 which is how i managed to run
the south migrations in the first place.
I upgraded to django 1.7 and then upon your request to 1.8, removed the
".id" part and i was able to regenerate the issue on 1.8.2 as well.
As for Tim's comment, Shaib's assumptions are correct. My
"content_scraper" app has no models, it just provides a "default" data
migration for the models provided by "dynamic_scraper".
Let me know how i can be of any further help.
Thanks,
Rakan
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:4>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
You could try adding an initial migration to the dynamic_scraper app (use
[https://docs.djangoproject.com/en/1.8/ref/settings/#migration-modules
settings.MIGRATION_MODULES]). Then update your data migration to have a
dependency on that migration.
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:5>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by rakanalh):
That worked.
I ran:
{{{
python manage.py makemigrations dynamic_scraper
}}}
Then
{{{
python manage.py migrate dynamic_scraper --fake
}}}
Because the tables already exist.
And ran migrate again on my app and everything ran as expected.
Could you please provide more context of why i needed to create a
migration for this third party app?
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:6>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
I think the problem is roughly described in
[https://docs.djangoproject.com/en/1.8/topics/migrations/#dependencies the
dependencies section] of the migrations docs. Basically, migrations and
models without migrations don't interact very well.
I don't think we are going to invest any time in trying to remedy this as
migrations will be compulsory for all apps in 1.9. We could add a sentence
to the 1.8 docs in the dependencies section, "In addition, any models that
are used in RunPython operations must have migrations."
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:7>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"df6a4cac52b58e471168d0a80e5d1900126b7154" df6a4cac]:
{{{
#!CommitTicketReference repository=""
revision="df6a4cac52b58e471168d0a80e5d1900126b7154"
[1.8.x] Refs #24853 -- Documented a limitation of RunPython and unmigrated
apps.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:8>
Django
-------------------------------------+-------------------------------------
Type: Uncategorized | Status: closed
Component: Database layer | Version: 1.7
(models, ORM) |
Keywords: database, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => wontfix
--
Ticket URL: <https://code.djangoproject.com/ticket/24853#comment:9>