Problems with dumpdata/loaddata

924 views
Skip to first unread message

jorr...@gmail.com

unread,
Apr 5, 2016, 7:45:44 PM4/5/16
to Django users
I want to dump and load data for two apps that I have. Using dumpdata works fine:

python manage.py dumpdata members posts > data.json

or

python manage.py dumpdata members posts --format=xml > data.xml

However, when I try to load the same data back with loaddata I get the following errors:

With JSON:
Traceback (most recent call last):
 
File "manage.py", line 10, in <module>
    execute_from_command_line
(sys.argv)
 
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 353, in ex
    utility
.execute()
 
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 345, in ex
   
self.fetch_command(subcommand).run_from_argv(self.argv)
 
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 348, in run_fr
   
self.execute(*args, **cmd_options)
 
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 399, in execut
    output
= self.handle(*args, **options)
 
File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 6
   
self.loaddata(fixture_labels)
 
File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 1
   
self.load_label(fixture_label)
 
File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 1
   
for obj in objects:
 
File "C:\Python27\lib\site-packages\django\core\serializers\json.py", line 76, in Deseri
    stream_or_string
= stream_or_string.decode('utf-8')
 
File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
   
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte



With XML:
Traceback (most recent call last):
 
File "manage.py", line 10, in <module>
    execute_from_command_line
(sys.argv)
 
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 353, in execute_from_comman
    utility
.execute()
 
File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 345, in execute
   
self.fetch_command(subcommand).run_from_argv(self.argv)
 
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 348, in run_from_argv
   
self.execute(*args, **cmd_options)
 
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 399, in execute
    output
= self.handle(*args, **options)
 
File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 60, in handle
   
self.loaddata(fixture_labels)
 
File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 100, in loaddata
   
self.load_label(fixture_label)
 
File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 152, in load_label
   
for obj in objects:
 
File "C:\Python27\lib\site-packages\django\utils\six.py", line 558, in next
   
return type(self).__next__(self)
 
File "C:\Python27\lib\site-packages\django\core\serializers\xml_serializer.py", line 174, in __next__
   
for event, node in self.event_stream:
 
File "C:\Python27\lib\xml\dom\pulldom.py", line 232, in next
    rc
= self.getEvent()
 
File "C:\Python27\lib\xml\dom\pulldom.py", line 265, in getEvent
   
self.parser.feed(buf)
 
File "C:\Python27\lib\xml\sax\expatreader.py", line 217, in feed
   
self._err_handler.fatalError(exc)
 
File "C:\Python27\lib\xml\sax\handler.py", line 38, in fatalError
   
raise exception
xml
.sax._exceptions.SAXParseException: <unknown>:1:31: encoding specified in XML declaration is incorrect

What gives?

ludovic coues

unread,
Apr 6, 2016, 8:21:35 AM4/6/16
to django...@googlegroups.com
Both error look like encoding error.
In the JS file, it look like there is a byte order mark at the start
of the file.
The first line of both file might help to understand the error.
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/dc3c9375-a180-45cb-b718-f272181575be%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Cordialement, Coues Ludovic
+336 148 743 42

Vijay Khemlani

unread,
Apr 6, 2016, 9:04:49 AM4/6/16
to django...@googlegroups.com
Your apps are called members and posts, or those are the names of the models?

jorr...@gmail.com

unread,
Apr 6, 2016, 10:10:23 AM4/6/16
to Django users
Thank you for the replies. members and posts are the names of my apps.

The first line of the XML file when I open it in Notepad says:

<?xml version="1.0" encoding="utf-8"?>

The JSON file has no distinguishable first line, it just starts right away with the data.

ludovic coues

unread,
Apr 7, 2016, 9:14:32 AM4/7/16
to django...@googlegroups.com
I believe the json file start with 0xff 0xfe, which mark the file as
UTF-16. The problem is that these two character are invalid at the
beginning of a json file.
The problem have the same root for the xml. UTF-16 file but it pretend
it is utf-8 encoded.

These problems might go away on another OS or with python 3.
Another solution might be to open the file with notepad++ and save as
utf-8, without BOM.
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/56a3a584-8052-4c80-bba9-35c4ae55cb96%40googlegroups.com.

jorr...@gmail.com

unread,
Apr 7, 2016, 8:17:05 PM4/7/16
to Django users
Opening the JSON file in Notepad++ certainly gives some insight... It says it's encoded in USC-2 LE BOM. Converting it to UTF-8 with BOM in Notepad++ solves the problem.

Question though, why does dumpdata create files with an encoding that can not be used out of the box with loaddata?

Ramiro Morales

unread,
Apr 8, 2016, 4:23:43 AM4/8/16
to django...@googlegroups.com

It shouldn't. Are you sure you haven't opened and saved the json file with a text editor that might be adding the BOM, e.g. Notepad before running loaddata?

On Apr 7, 2016 9:17 PM, <jorr...@gmail.com> wrote:
Opening the JSON file in Notepad++ certainly gives some insight... It says it's encoded in USC-2 LE BOM. Converting it to UTF-8 with BOM in Notepad++ solves the problem.

Question though, why does dumpdata create files with an encoding that can not be used out of the box with loaddata?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

jorr...@gmail.com

unread,
Apr 9, 2016, 9:50:13 AM4/9/16
to Django users
I jus tested it again and it happens even without opening the file in an editor first. Would that be a bug then?

Matthijs Kooijman

unread,
Nov 6, 2017, 1:21:05 PM11/6/17
to Django users
Hey folks,

I'm replying to an old thread about a problem with "manage.py loaddata" generating a BOM (byte-order-mark) that "manage.py dumpdata" refuses to load. I just ran into this same problem and found that, in my case, the problem is not "loaddata", but powershell which I was running under. Apparently powershell adds a BOM when redirecting output. For example, a simple "echo foo > somefile" also adds a BOM. Seeing the original poster was also using Windows, perhaps he was also using powershell (the regular Windows cmd.exe does not seem to have this behaviour).

For me, the solution was to just use a different shell, or use the "--output" option to dumpdata.

Gr.

Matthijs

Jani Tiainen

unread,
Nov 6, 2017, 2:46:54 PM11/6/17
to django...@googlegroups.com
Hi.

Maybe it would be useful to create ticket to document this behavior in Django documentation.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
Reply all
Reply to author
Forward
0 new messages