[Django] #26721: "loaddata" can not load data generated by "dumpdata" due to encoding issues
Django
--------------------------------------+-------------------------------
Reporter: OBu | Owner: nobody
Type: Bug | Status: new
Component: Core (Serialization) | Version: 1.9
Severity: Normal | Keywords: dumpdata loaddata
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+-------------------------------
When I "dumpdata" my database on my Windows 8.1 machine and load the
dumped file with "loaddata", I get encoding errors.
Opening the dumped file with ntepad++, setting the encoding to UTF-8 and
saviong it again helps.
http://python.6.x6.nabble.com/Problems-with-dumpdata-loaddata-
td5187942.html
describes the problem in detail so it seems I'm not alone...
It might not be a Django problem but a windows pipeline-issue (I was using
the powershell), but it is a bit embarassing.
--
Ticket URL: <https://code.djangoproject.com/ticket/26721>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: OBu | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
Keywords: dumpdata loaddata | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Old description:
> When I "dumpdata" my database on my Windows 8.1 machine and load the
> dumped file with "loaddata", I get encoding errors.
>
> Opening the dumped file with ntepad++, setting the encoding to UTF-8 and
> saviong it again helps.
>
> http://python.6.x6.nabble.com/Problems-with-dumpdata-loaddata-
> td5187942.html
> describes the problem in detail so it seems I'm not alone...
>
> It might not be a Django problem but a windows pipeline-issue (I was
> using the powershell), but it is a bit embarassing.
New description:
When I "dumpdata" my database on my Windows 8.1 machine and load the
dumped file with "loaddata", I get encoding errors.
Opening the dumped file with ntepad++, setting the encoding to UTF-8 and
saving it again helps.
[https://groups.google.com/d/topic/django-users/NAHD058Gh_Q/discussion A
django-users thread] describes the problem in detail so it seems I'm not
alone...
It might not be a Django problem but a windows pipeline-issue (I was using
the powershell), but it is a bit embarrassing.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:1>
Django
Reporter: OBu | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Easy pickings: 0 | UI/UX: 0
Comment (by bmispelon):
Hi,
Are you able to try to do `dumpdata` with both Python 2 and Python 3? Does
it make any difference?
Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:2>
Django
Reporter: OBu | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Easy pickings: 0 | UI/UX: 0
Comment (by OBu):
Sorry, I don't. I was using python 3.5.1 and django 1.9.7
I just tried `dir > test.txt` in my windows command line, just to check
the encoding of the resulting file, and notepad++ tells me it is UTF-8.
When I do the same in Windows Powershell, encoding is UCE-2 BE BOM
I assume this is exactly what happened to me, so it's not really a Django
issue but a problem resulting from the character encoding in Windwos
Powershell.
I'm not sure whether there is a way to fix this other then using the
`--output` option (which I did not test), but maybe this problem should be
mentioned in the docs.
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:3>
Django
Reporter: OBu | Owner: nobody
Type: Bug | Status: new
(Serialization) |
Severity: Normal | Resolution:
| Unreviewed
Easy pickings: 0 | UI/UX: 0
Description changed by OBu:
Old description:
> When I "dumpdata" my database on my Windows 8.1 machine and load the
> dumped file with "loaddata", I get encoding errors.
>
> Opening the dumped file with ntepad++, setting the encoding to UTF-8 and
> saving it again helps.
>
> [https://groups.google.com/d/topic/django-users/NAHD058Gh_Q/discussion A
> django-users thread] describes the problem in detail so it seems I'm not
> alone...
>
> It might not be a Django problem but a windows pipeline-issue (I was
> using the powershell), but it is a bit embarrassing.
New description:
When I "dumpdata" my database on my Windows 8.1 machine and load the
dumped file with "loaddata", I get encoding errors.
Opening the dumped file with notepad++, setting the encoding to UTF-8 and
saving it again helps.
[https://groups.google.com/d/topic/django-users/NAHD058Gh_Q/discussion A
django-users thread] describes the problem in detail so it seems I'm not
alone...
It might not be a Django problem but a windows pipeline-issue (I was using
the powershell), but it is a bit embarrassing.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:4>
Django
Windows PowerShell
--------------------------------------+------------------------------------
Reporter: OBu | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 1.9
Severity: Normal | Resolution:
Keywords: dumpdata loaddata | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* component: Core (Serialization) => Documentation
* stage: Unreviewed => Accepted
* type: Bug => Cleanup/optimization
Comment:
I'm not knowledgeable about Windows to confirm, but assuming some can
confirm it, a documentation note seems okay.
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:5>
Django
Windows PowerShell
--------------------------------------+------------------------------------
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 1.9
Severity: Normal | Resolution:
Keywords: dumpdata loaddata | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson):
#32439 was closed as a duplicate, with `dumpdata` failing due to the
target file not explicitly setting the encoding.
Likely the solution here is to add to the
[https://docs.djangoproject.com/en/3.1/howto/windows/ Windows How-To] that
the default encoding can be set UTF8 now:
> Go to Settings there's a "Use Unicode UTF-8 for worldwide language
support", box in "Language" - "Administrative Language Settings" - "Change
system locale" - "Region Settings".
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:6>
Django
Windows PowerShell
Reporter: OBu | Owner: David
Type: | Smith
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 1.9
Severity: Normal | Resolution:
Keywords: dumpdata loaddata | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by David Smith):
* owner: nobody => David Smith
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:7>
Django
Windows PowerShell
-------------------------------------+-------------------------------------
Reporter: OBu | Owner: David
Type: | Smith
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 1.9
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by David Smith):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:8>
Django
Windows PowerShell
-------------------------------------+-------------------------------------
Reporter: OBu | Owner: David
Type: | Smith
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 1.9
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:9>
Django
Windows PowerShell
-------------------------------------+-------------------------------------
Reporter: OBu | Owner: David
Type: | Smith
Component: Documentation | Version: 1.9
Severity: Normal | Resolution: fixed
Keywords: dumpdata loaddata | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by GitHub <noreply@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"0456d3e42795481a186db05719300691fe2a1029" 0456d3e4]:
{{{
#!CommitTicketReference repository=""
revision="0456d3e42795481a186db05719300691fe2a1029"
Fixed #26721 -- Doc'd setting UTF-8 on Windows.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:10>
Django
Windows PowerShell
-------------------------------------+-------------------------------------
Reporter: OBu | Owner: David
Type: | Smith
Cleanup/optimization | Status: closed
Component: Documentation | Version: 1.9
Severity: Normal | Resolution: fixed
Keywords: dumpdata loaddata | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"d6b6eda4ed3193745a8d54bf364e5ecf41b7862e" d6b6eda4]:
{{{
#!CommitTicketReference repository=""
revision="d6b6eda4ed3193745a8d54bf364e5ecf41b7862e"
[3.2.x] Fixed #26721 -- Doc'd setting UTF-8 on Windows.
Backport of 0456d3e42795481a186db05719300691fe2a1029 from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26721#comment:11>