[Django] #29249: Add option to dumpdata with unicode data
Django
---------------------------------------+-----------------------------------
Reporter: hakib | Owner: nobody
Type: New feature | Status: new
Component: Utilities | Version: 2.0
Severity: Normal | Keywords: dumpdata, unicode
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------+-----------------------------------
It is currently not (easily) possible to use the `dumpdata` managemet
command on models with unicode data.
The JSON serializer used by dumpdata is not accepting the `ensure_ascii`
argument used by `json.dumps` as an argument.
Since `ensure_ascii=True` is the default, I suggest adding a `--dont-
ensure-ascii` flag to the `dumpdata` managemet command so it will be
easier to use dumpdata with unicode.
{{{
./manage.py dumpdata app.model --dont-ensure-ascii
}}}
I'm not sure what are the implications on other serializers such as YAML,
XML etc.
--
Ticket URL: <https://code.djangoproject.com/ticket/29249>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: nobody
Type: New feature | Status: new
commands) |
Severity: Normal | Resolution:
Keywords: dumpdata, unicode | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* component: Utilities => Core (Management commands)
Comment:
I believe this would only apply to the JSON serializer, and I'm not sure
about adding a dumpdata option that's specific to a particular serializer.
I think your best solution is to subclass the JSON serializer,
[https://docs.djangoproject.com/en/dev/ref/settings/#serialization-modules
register it as a custom format], and then use that format in `dumpdata`.
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: nobody
Type: New feature | Status: new
commands) |
Severity: Normal | Resolution:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by hakib):
The JSON serializer has a `ensure_ascii` attribute and the YAML serializer
has a `allow_unicode` attribute. I already sumitted a
[PR](https://github.com/django/django/pull/9818#issuecomment-375694612)
implemeting the flag in both serializers.
I haven't looked at the XML serializer yet but i'm sure it will be
possible there as well.
As someone who works with unicode as the primary language for most apps
(as i'm sure a lot of other developers do) it's a very a usefull feature
to be able to dump fixtures directly from local db in a readbale format.
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: nobody
Type: New feature | Status: new
commands) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* needs_better_patch: 0 => 1
* has_patch: 0 => 1
* stage: Unreviewed => Accepted
Comment:
Okay. My main concern is that an option calls `--allow_unicode` may
suggest to readers that all serializers prohibit unicode by default. That
may not be true. Your patch also needs documentation.
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: nobody
Type: New feature | Status: new
commands) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:4>
Django
-------------------------------------+-------------------------------------
| Ramezani
Type: New feature | Status: assigned
Component: Core (Management | Version: 2.0
commands) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Hasan Ramezani):
* owner: nobody => Hasan Ramezani
* needs_better_patch: 1 => 0
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: Hasan
| Ramezani
Type: New feature | Status: assigned
commands) |
Severity: Normal | Resolution:
Keywords: dumpdata, unicode | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* has_patch: 1 => 0
* version: 2.0 => master
Comment:
Current behavior is inconsistent. XML serializer use Unicode by default,
on the other hand YAML and JSON serializers force ASCII. I think we should
make this behavior consistent instead of adding a new serializer-specific
option, i.e. pass `allow_unicode=True` to `yaml.dump()` and
`ensure_ascii=False` to `json.dump()`.
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: Hasan
| Ramezani
Type: New feature | Status: assigned
Component: Core (Management | Version: master
commands) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Hasan Ramezani):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: Hasan
| Ramezani
Type: New feature | Status: assigned
Component: Core (Management | Version: master
commands) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: Hasan
| Ramezani
Type: New feature | Status: assigned
Component: Core (Management | Version: master
commands) |
Severity: Normal | Resolution:
Keywords: dumpdata, unicode | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"8970bb4cfdc56cb90b84ff2154afe622c9e9eaf7" 8970bb4]:
{{{
#!CommitTicketReference repository=""
revision="8970bb4cfdc56cb90b84ff2154afe622c9e9eaf7"
Refs #29249 -- Added tests for serializing Unicode data with XML
serializer.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:9>
Django
-------------------------------------+-------------------------------------
Reporter: hakib | Owner: Hasan
| Ramezani
Component: Core (Management | Version: master
commands) |
Keywords: dumpdata, unicode | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"68fc21b3784aa34c7ba5515ab02ef0c7b6ee856d" 68fc21b]:
{{{
#!CommitTicketReference repository=""
revision="68fc21b3784aa34c7ba5515ab02ef0c7b6ee856d"
Fixed #29249 -- Made JSON and YAML serializers use Unicode by default.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29249#comment:10>