[Django] #21075: django crashes with sys.exit(1) when management.call_command is used with unittest
Django
unittest
--------------------------------------+--------------------
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: new
Component: Core (Serialization) | Version: 1.5
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+--------------------
Hi Everyone,
I would like to created a backup of my models when new model elements are
created, therefore I
defined in my `models.py` the following function:
{{{
def save_models_as_json(**kwargs):
with open(JSON_DATA_FILE, 'w') as j:
management.call_command('dumpdata', 'TimePortal',
indent=4, stdout=j)
# later in my models file I register it with:
post_save.connect(save_models_as_json, sender=Employee)
}}}
The above command is working perfectly fine when I use the admin GUI I
find
inside the `JSON_DATA_FILE` all my relevand information from my
application
TimePortal.
This function also works fine when running unit tests with:
{{{
$ python mange test TimePortal
}}}
Things get ugly when I want the dumpdata command to include some more
arguments, e.g:
{{{
def save_models_as_json(**kwargs):
with open(JSON_DATA_FILE, 'w') as j:
management.call_command('dumpdata', 'TimePortal', '--natural'
indent=4, stdout=j)
}}}
or:
{{{
def save_models_as_json(**kwargs):
with open(JSON_DATA_FILE, 'w') as j:
management.call_command('dumpdata', 'TimePortal', '-e',
'contenttypes',
indent=4, stdout=j)
}}}
When I run my unit tests, django will crash, with either:
{{{
$ python ../manage.py test TimePortal
Creating test database for alias 'default'...
........Error: Unknown application: --natural
E............
======================================================================
ERROR: test_customer_has_employee (TimePortal.test_models.TestModels)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/ozn/software/zeitport/zeitportal/TimePortal/test_models.py", line
18, in setUp
worksfor=self.customer)
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/manager.py", line 134, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/query.py", line 452, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/base.py", line 463, in save
self.save_base(using=using, force_insert=force_insert,
force_update=force_update)
File "/usr/local/lib/python2.7/dist-
packages/django/db/models/base.py", line 565, in save_base
created=(not record_exists), raw=raw, using=using)
File "/usr/local/lib/python2.7/dist-
packages/django/dispatch/dispatcher.py", line 172, in send
response = receiver(signal=self, sender=sender, **named)
File "/home/ozn/software/zeitport/zeitportal/TimePortal/models.py",
line 61, in save_models_as_json
'dumpdata', '--natural', 'TimePortal', indent=4, stdout=j)
File "/usr/local/lib/python2.7/dist-
packages/django/core/management/__init__.py", line 150, in call_command
return klass.execute(*args, **defaults)
File "/usr/local/lib/python2.7/dist-
packages/django/core/management/base.py", line 249, in execute
sys.exit(1)
SystemExit: 1
----------------------------------------------------------------------
Ran 21 tests in 0.017s
FAILED (errors=1)
Destroying test database for alias 'default'...
}}}
or with
{{{
$ python ../manage.py test TimePortal
Creating test database for alias 'default'...
........Error: Unknown application: -e
E............
======================================================================
... chopped for sanity ...
File "/usr/local/lib/python2.7/dist-
packages/django/core/management/__init__.py", line 150, in call_command
return klass.execute(*args, **defaults)
File "/usr/local/lib/python2.7/dist-
packages/django/core/management/base.py", line 249, in execute
sys.exit(1)
SystemExit: 1
----------------------------------------------------------------------
Ran 21 tests in 0.017s
FAILED (errors=1)
Destroying test database for alias 'default'...
}}}
Thanks for the excellent django documentation, I found out I could
acchieve
the desired serialization without using `management.call_command`:
{{{
from django.core import serializers
JSONSerializer = serializers.get_serializer('json')
json_serializer = JSONSerializer()
def save_models_as_json(**kwargs):
with open(JSON_DATA_FILE, 'w') as j:
json_serializer.serialize(list(Employee.objects.all())
+ list(Customer.objects.all())
+list(Rules.objects.all())
+ list(Account.objects.all()),
indent=4, use_natural_keys=True)
j.writelines(json_serializer.getvalue())
}}}
despite the above 'solution', I think this is worth the attention of the
developers, since it might give a hint for some problem with
management.call_command.
I am using django 1.5.2 and Python 2.7.5.
I also tested this with Python 2.6.8 and django 1.4.5
Thanks for reading so far. Hopefully, you can find a good explanation to
this behavior with multiple options passed to `management.call_command`.
--
Ticket URL: <https://code.djangoproject.com/ticket/21075>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: closed
Component: Core | Version: 1.5
(Serialization) | Resolution: invalid
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by bmispelon):
* status: new => closed
* needs_better_patch: => 0
* resolution: => invalid
* needs_tests: => 0
* needs_docs: => 0
Comment:
Hi,
This looks like a duplicate of #18387, which was fixed in 1.5.
Can you double-check that you can reproduce the issue in 1.5 and provide a
traceback (the one you gave is 1.4 if I'm not mistaken)?
I'll close the issue in the meantime. Please reopen it if you have some
more information.
Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:1>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: new
(Serialization) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by nahumoz@…):
* status: closed => new
* resolution: invalid =>
Comment:
Hi,
Here is a full transcript of actions with Python 2.7.5 and Djang 1.5.2:
{{{
[11] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [master] $ git
stash
Saved working directory and index state WIP on master: 361cd0b fixes for
formatting strings in python 2.6 vs. 2.7
HEAD is now at 361cd0b fixes for formatting strings in python 2.6 vs. 2.7
[12] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [master] $ git
checkout buggy
Switched to branch 'buggy'
[13] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $ which
python
/usr/bin/python
[14] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $
python -V
Python 2.7.5+
[15] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $
python -c 'import django; print(django.get_version())'
1.5.2
[16] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $
python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -e
[17] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $ cat
models.py
# -*- coding: UTF-8 -*-
from __future__ import absolute_import
from django.db import models
from django.contrib.auth.models import User
#
#
#
class File(models.Model):
"""
the database model for the time portal application
we store:
- the uploaded file content
- the date of the check
- the uploader
- the date of the processing
- the processor
"""
zeitFile = models.CharField(max_length=200000)
comment = models.CharField(max_length=1000)
checkDate = models.DateTimeField('date checked')
uploadBy = models.CharField(max_length=100)
processedDate = models.DateTimeField('date processed', null=True)
processedMarked = models.CharField(max_length=10, null=True)
processedBy = models.CharField(max_length=100, null=True)
#
#
#
#
from django.db.models.signals import post_save
from django.core import management
JSON_DATA_FILE = '/tmp/TimePortal.json'
def save_models_as_json(**kwargs):
with open(JSON_DATA_FILE, 'w') as j:
management.call_command('dumpdata', 'TimePortal', '-e',
'contenttypes',
indent=4, stdout=j)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:2>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: closed
Component: Core | Version: 1.5
(Serialization) | Resolution: invalid
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by bmispelon):
* status: new => closed
* resolution: => invalid
Comment:
Hi,
This is not a bug, and the error message is quite clear:
> CommandError: Unknown application: -e
You need to put the options before the appname, as indicated by `manage.py
help dumpdata`:
> Usage: manage.py dumpdata [options] [appname appname.ModelName ...]
Hope that helps.
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:3>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: new
(Serialization) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by nahumoz@…):
* status: closed => new
* resolution: invalid =>
Comment:
I am sorry, but it crashes also if I change the order.
{{{
[24] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [master] $ git
checkout buggy
Switched to branch 'buggy'
[25] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $ vim
models.py
[26] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $
python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -e
[27] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $ grep
command models.py
management.call_command('dumpdata', '-e', 'contenttypes',
'TimePortal',
}}}
The function is now:
{{{
def save_models_as_json(**kwargs):
with open(JSON_DATA_FILE, 'w') as j:
management.call_command('dumpdata', '-e', 'contenttypes',
'TimePortal',
indent=4, stdout=j)
}}}
and even this is not strictly a `bug` the error message is not quite
clear.
It did happen also with other parameters :
{{{
[33] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $
python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -n
[34] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $ man
cat
[35] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $ grep
command models.py
management.call_command('dumpdata', '-n', 'TimePortal',
[36] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:4>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: closed
Component: Core | Version: 1.5
(Serialization) | Resolution: invalid
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by bmispelon):
* cc: bmispelon@… (added)
* status: new => closed
* resolution: => invalid
Comment:
I had a look at the documentation for `call_command` [1] and figured out
what was wrong.
You're not supposed to pass options like `-e` as positional arguments but
as named arguments instead, like so:
{{{
call_command('dumpdata', 'TimePortal', exclude='contenttypes', indent=4,
stdout=j)
}}}
I agree that the error message is not completely clear (though it does
tell you that the command is trying to interpret "-e" as an application
name), but I think the documentation is helpful and the examples are
pretty straightforward.
[1] https://docs.djangoproject.com/en/1.5/ref/django-
admin/#django.core.management.call_command
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:5>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: new
(Serialization) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by anonymous):
* status: closed => new
* resolution: invalid =>
Comment:
Sorry, this is still an issue ... it does not crash with sys.exit(1). But
the documentation is just lacking and your example does not work:
{{{
$ grep contenttypes models.py
management.call_command('dumpdata', 'TimePortal',
exclude='django.contrib.contenttypes', indent=4, stdout=j)
[21] ozn@deboz:~/software/zeitport/zeitportal/TimePortal [buggy] $
python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown app in excludes: d
}}}
tried here with `exclude='django.contrib.contenttypes'` because
`exclude='contenttypes'` gave the error:
{{{
CommandError: Unknown app in excludes: d
}}}
with `exclude='natural'` because `natural` gave the error:
{{{
CommandError: Unknown application: natural
}}}
Could you please check this?
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:6>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: new
(Serialization) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by nahumoz@…):
oops, posted wrong info.
about the 'natural' the line was simply:
{{{
management.call_command('dumpdata', 'TimePortal',
exclude='django.contrib.contenttypes', indent=4, stdout=j)
}}}
Sorry about that.
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:7>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: new
(Serialization) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by bmispelon):
Looks like `exclude` is expecting a list of apps to exclude.
Try `exclude=['contenttypes']` instead.
About the documentation, what do you find lacking? Maybe we can improve it
with your feedback.
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:8>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: new
(Serialization) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by nahumoz@…):
phew, thanks for not giving up on me. I really appreciate it.
Well, for example that some keywords expect a list, and that that other
are booleans, e.g.:
{{{
management.call_command('dumpdata', 'TimePortal',
exclude=['contenttypes'], use_natural_keys=True)
}}}
To someone who does not know all the ins and outs of argparse and
django.core.management. It is not that trivial that the above in code call
is the transfomation of the following shell command*:
{{{
$ python manage.py dumpdata -e contenttypes -natural TimePortal
}}}
I could probably figure out all
of that by reading the source code itself, but I guess documentation is
there to save everyone that useful time.
* btw. in the shell, argparse does not care about order, so this works
too:
{{{
python ../manage.py dumpdata -n -e contenttypes TimePortal
}}}
The method `call_command` does care about order.
So maybe you could include all these insights under this section
https://docs.djangoproject.com/en/1.5/ref/django-
admin/#django.core.management.call_command.
Cheers, Oz
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:9>
Django
unittest
Reporter: nahumoz@… | Owner: nobody
Type: Bug | Status: new
(Serialization) | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by nahumoz@…):
Hi Baptiste,
You were very helpful and it encouraged me to contribute back. Thank you!
Please see this request:
https://github.com/django/django/pull/1609
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:10>
Django
--------------------------------------+------------------------------------
Reporter: nahumoz@… | Owner: timo
Type: Cleanup/optimization | Status: assigned
Component: Documentation | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timo):
* status: new => assigned
* component: Core (Serialization) => Documentation
* version: 1.5 => master
* owner: nobody => timo
* has_patch: 0 => 1
* type: Bug => Cleanup/optimization
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:11>
Django
--------------------------------------+------------------------------------
Reporter: nahumoz@… | Owner: timo
Component: Documentation | Version: master
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"fca4c4826e7b4cec84c3f8140bb929e38eea962c"]:
{{{
#!CommitTicketReference repository=""
revision="fca4c4826e7b4cec84c3f8140bb929e38eea962c"
Fixed #21075 - Improved doc for calling call_command with arguments.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:12>
Django
--------------------------------------+------------------------------------
Reporter: nahumoz@… | Owner: timo
Type: Cleanup/optimization | Status: closed
Component: Documentation | Version: master
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"960f5bc75901f76e9b4b356f98b22b494fb47611"]:
{{{
#!CommitTicketReference repository=""
revision="960f5bc75901f76e9b4b356f98b22b494fb47611"
[1.6.x] Fixed #21075 - Improved doc for calling call_command with
arguments.
Backport of fca4c4826e from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:13>
Django
--------------------------------------+------------------------------------
Reporter: nahumoz@… | Owner: timo
Type: Cleanup/optimization | Status: closed
Component: Documentation | Version: master
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"693ebff1a4f5eeda56279188ee473be6a1088e46"]:
{{{
#!CommitTicketReference repository=""
revision="693ebff1a4f5eeda56279188ee473be6a1088e46"
[1.5.x] Fixed #21075 - Improved doc for calling call_command with
arguments.
Backport of fca4c4826e from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/21075#comment:14>