Problems with dumpdata/loaddata

[jorr...@gmail.com]

I want to dump and load data for two apps that I have. Using dumpdata works fine:

python manage.py dumpdata members posts > data.json

or

python manage.py dumpdata members posts --format=xml > data.xml

However, when I try to load the same data back with loaddata I get the following errors:

With JSON:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 353, in ex
    utility.execute()
  File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 345, in ex
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Python27\lib\site-packages\django\core\management\base.py", line 348, in run_fr
    self.execute(*args, **cmd_options)
  File "C:\Python27\lib\site-packages\django\core\management\base.py", line 399, in execut
    output = self.handle(*args, **options)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 6
    self.loaddata(fixture_labels)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 1
    self.load_label(fixture_label)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 1
    for obj in objects:
  File "C:\Python27\lib\site-packages\django\core\serializers\json.py", line 76, in Deseri
    stream_or_string = stream_or_string.decode('utf-8')
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte

With XML:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 353, in execute_from_comman
    utility.execute()
  File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Python27\lib\site-packages\django\core\management\base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\Python27\lib\site-packages\django\core\management\base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 60, in handle
    self.loaddata(fixture_labels)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 100, in loaddata
    self.load_label(fixture_label)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 152, in load_label
    for obj in objects:
  File "C:\Python27\lib\site-packages\django\utils\six.py", line 558, in next
    return type(self).__next__(self)
  File "C:\Python27\lib\site-packages\django\core\serializers\xml_serializer.py", line 174, in __next__
    for event, node in self.event_stream:
  File "C:\Python27\lib\xml\dom\pulldom.py", line 232, in next
    rc = self.getEvent()
  File "C:\Python27\lib\xml\dom\pulldom.py", line 265, in getEvent
    self.parser.feed(buf)
  File "C:\Python27\lib\xml\sax\expatreader.py", line 217, in feed
    self._err_handler.fatalError(exc)
  File "C:\Python27\lib\xml\sax\handler.py", line 38, in fatalError
    raise exception
xml.sax._exceptions.SAXParseException: <unknown>:1:31: encoding specified in XML declaration is incorrect

What gives?

[ludovic coues]

Both error look like encoding error.
In the JS file, it look like there is a byte order mark at the start
of the file.
The first line of both file might help to understand the error.

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/dc3c9375-a180-45cb-b718-f272181575be%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Cordialement, Coues Ludovic
+336 148 743 42

[Vijay Khemlani]

Your apps are called members and posts, or those are the names of the models?

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAEuG%2BTaNgXH0hUrc-mVpK%2BFe79UXj76gAL33Ozw7Fe25LEbQ%3Dg%40mail.gmail.com.

[jorr...@gmail.com]

Thank you for the replies. members and posts are the names of my apps.

The first line of the XML file when I open it in Notepad says:

<?xml version="1.0" encoding="utf-8"?>

The JSON file has no distinguishable first line, it just starts right away with the data.

[ludovic coues]

I believe the json file start with 0xff 0xfe, which mark the file as
UTF-16. The problem is that these two character are invalid at the
beginning of a json file.
The problem have the same root for the xml. UTF-16 file but it pretend
it is utf-8 encoded.

These problems might go away on another OS or with python 3.
Another solution might be to open the file with notepad++ and save as
utf-8, without BOM.

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/56a3a584-8052-4c80-bba9-35c4ae55cb96%40googlegroups.com.

[jorr...@gmail.com]

Opening the JSON file in Notepad++ certainly gives some insight... It says it's encoded in USC-2 LE BOM. Converting it to UTF-8 with BOM in Notepad++ solves the problem.

Question though, why does dumpdata create files with an encoding that can not be used out of the box with loaddata?

[Ramiro Morales]

It shouldn't. Are you sure you haven't opened and saved the json file with a text editor that might be adding the BOM, e.g. Notepad before running loaddata?

On Apr 7, 2016 9:17 PM, <jorr...@gmail.com> wrote:

Opening the JSON file in Notepad++ certainly gives some insight... It says it's encoded in USC-2 LE BOM. Converting it to UTF-8 with BOM in Notepad++ solves the problem.

Question though, why does dumpdata create files with an encoding that can not be used out of the box with loaddata?

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b3f4fbbf-bcad-4376-8ed1-aa476b8c8a88%40googlegroups.com.

[jorr...@gmail.com]

I jus tested it again and it happens even without opening the file in an editor first. Would that be a bug then?

[Matthijs Kooijman]

Hey folks,

I'm replying to an old thread about a problem with "manage.py loaddata" generating a BOM (byte-order-mark) that "manage.py dumpdata" refuses to load. I just ran into this same problem and found that, in my case, the problem is not "loaddata", but powershell which I was running under. Apparently powershell adds a BOM when redirecting output. For example, a simple "echo foo > somefile" also adds a BOM. Seeing the original poster was also using Windows, perhaps he was also using powershell (the regular Windows cmd.exe does not seem to have this behaviour).

For me, the solution was to just use a different shell, or use the "--output" option to dumpdata.

Gr.

Matthijs

[Jani Tiainen]

Hi.

Maybe it would be useful to create ticket to document this behavior in Django documentation.

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/227a6a31-4256-49a8-a11b-704b26d58dd3%40googlegroups.com.