Re: [Django Code] #2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in mysql)

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------
Comment (by al...@halogen-dg.com):

Hello,

I have looked at the various patches and tried to produce my own, improved
version. I've improved it by by adding modeltests, made sure it works fine
with MySQL, PostgresSQL and SQLite.

The main idea here is to get similar results, similar data types for all
supported databases - that is why I used 'buffer' type to
be always returned from backend.

I have re-read all related discussions in google groups, and looked at
tickets 2422, 5135.

A few small problems described in django-developers and other bugs has
been solved too, like using 'repr' instead of 'unicode'to show SQL debug
in Debug mode.

I am ready to support this patch for some time until it gets stable and
good enough to have a chance being included to trunk.

Comments are welcome. I don't think that this work is 100% final, but it
have good chances to became good enough if you comment & help me to
improve.

We are using this patch in a production version of one site, and we will
improve it along the way.

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:41>
Django Code <http://code.djangoproject.com/>
The web framework for perfectionists with deadlines

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------

Changes (by anonymous):

* cc: al...@halogen-dg.com (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:42>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------

Comment (by mtredinnick):

Thanks for looking at the "make it work across all databases" aspect of
this. That's a necessary part of the process. I had a quick read of the
patch. I like the use of `buffer()`, although just using a bytestring
(Python's `str` type) would have worked to. We might end up using `str`
for simplicity in the end, but that's a very minor thing.

There are a few things that I don't immediately like, however (and I'm
really happy you've done the work here, so take this as a serious design
review as part of getting it into trunk, rather than dissing your effort):
it feels messy to have to test for certain field types in
`Model.__init__`. Part of that would be solved by using (or at least,
permitting) bytestrings instead of buffers. More generally, though, you've
introduced a problem with all other data types. Using `repr(v)` instead of
simply `v` in the `backends/__init__.py` file when you're converting to
unicode means that all the places where smart objects with a
`__unicode__` method are used for database input will no longer work. You
can't do that. I'm also a little surprised that change works, because
calling `unicode()` on something that isn't valid UTF-8 data (which binary
data can easily do) is just going to be broken there. What is you have
data such as `\xff\xff` for example, which isn't valid UTF-8?

Instead, I'd consider creating a special storage type (e.g. `BinaryData`)
analogous to `SafeString`. The database backend wrappers are then taught
to treat such data as binary, rather than converting them to/from unicode.
The conversion between `BinaryData` (or whatever it's called) and `str` or
`buffer` can be done in `db_prep_save()` and `db_prep_load()`, etc. I
haven't done all the research about what is needed to convince each
backend to treat the data as real binary data (rather than converting it
from UTF-8 to Unicode or something) and that was one thing holding up me
doing anything here. From your patch, it looks like the answer is that not
much is required, which is better than I'd hoped, but you have to do
something to not require the change in `backend/__init__.py`.

I'm a little concerned to see `get_manipulator_*` functions being added,
simply because they shouldn't be needed. Manipulators are Old Busted and
newforms-style fields and newforms-admin (which merged probably after you
made this patch, so it's no biggie) are the New Hotness. So nothing
manipulator-related should be needed.

Finally, it will be easier to review the patches if the random blank lines
and print statements around line 157 of `fields/__init.py` weren't there
acting as a distraction.

It's quite possible I'm missing some subtle stuff here and I'll think
about it a bit more to check that, but the above are the main things I'm
concerned about at the moment. Particularly, the leaking of binary data
into `Model.__init__` and the incorrect handling of converting objects to
Unicode strings in `backend/__init__.py`. If anything's not clear, or you
want to bounce ideas around in a better UI than a webform in Trac, feel
free to email me directly.

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:43>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------

Changes (by Mark Ng <dja...@markng.co.uk>):

* cc: dja...@markng.co.uk (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:44>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------

Comment (by al...@halogen-dg.com):

> Thanks for looking at the "make it work across all databases" aspect of
this

We also plan to install Oracle (may be) and try to test it with it.

> I like the use of buffer(), although just using a bytestring (Python's

str type) would have
> worked to.

buffer() has been used by performance reasons, PostgreSQL and SQLite
already return it (and psycopg2 developers mention that its a way faster
of using str here).

> it feels messy to have to test for certain field types in Model.__init__

Agree. We have found better solution, by using MySQL converters. Other
database backends don't require any conversion.

> Using repr(v) instead of simply v in the backends

This conversion was only applied when DEBUG=True and some SQL error raise.
Since binary data can be quite large, we have decided to print only
indication that binary data is passed, and size of it. May be thats not
ideal, but to understand problem in SQL should be enough.

Alex & Vic @ halogen-dg.com

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:45>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------
Comment (by al...@halogen-dg.com):

Please note, this patch does not yet include:
(a) Newform Admin functionality
(b) Fixtures dump/load

So its not quite ready for check-in. As soon as we finish all those
functional parts we will notify.

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:46>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------

Changes (by hann...@gmail.com):

* cc: hann...@gmail.com (added)

Comment:

Just a quick reminder that at least Postgres' BYTEA is deliberately used
to store raw bytestrings in legacy databases, aka. {{{b''}}} in Pythons-
to-be. Just thinking forward here... For instance, amavisd-new (antispam
system) now uses BYTEA instead VARCHAR and TEXT for all email-headers
since email-headers are all defined to be 7-bit ASCII and hence BYTEA is
more ideologically correct (Email with headers that are not 7-bit ASCII
are breaking the RFCs). This means that a frontend to amavisd-new (and
potentially any system that stores email in SQL) written in Django would
have to be able to dump the contents of a BYTEA-field as text. I foresee
that just uploads and downloads of binary data will lead to "how to edit
binary data in a text-field in the admin" to become a FAQ.

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:47>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------

Comment (by mspang):

What are the prospects of including this patch in 1.0? There are lots of
applications that could use this. F.e., any app that stores a hash like
SHA1 or a UUID would need this. Such apps may pass over django if 1.0
doesn't allow this. Using base64 is tedious, and you can't even work
around that using a custom field because there's no distinction between
the "load field from database" and "set field by user" operations in a
Field. You would never know in to_python() whether you were receiving a
base64 string or a binary string.

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:48>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------

Comment (by mtredinnick):

@mspang: Ticket comments are not the place to ask when something will be
included. That doesn't help move it towards resolution.

@hanne.moa: Being able to treat binary fields as text transparently isn't
a design requirement for this ticket. If you want to do that and don't
want to just do the buffer -> string conversion you can subclass the
eventual binary field class, or you can easily enough write your own
class. Django doesn't provide every possible db column type to Python type
and that's intentional. We aim to hit the common and most natural uses
(and storing text in a database naturally uses a text field, not a binary
field) and allow people to write their own field subclasses for extra
cases.

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:49>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------

Changes (by anonymous):

* cc: dcwa...@gmail.com (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:50>

[Django Code]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
------------------------------------------+---------------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database wrapper | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
------------------------------------------+---------------------------------
Changes (by anonymous):

* cc: sp...@google.com (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:51>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)

---------------------------------------------------+------------------------

Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:

Component: Database layer (models, ORM) | Version:

Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |

---------------------------------------------------+------------------------
Changes (by lidaobing):

* cc: lida...@gmail.com (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:52>
Django <http://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Changes (by ConradIrwin):

* cc: conrad...@gmail.com (added)

Comment:

patch.3.txt is not an improvement over patch.2.txt, just the context of
one of the changes had changed.

It would be really great if this could become part of django by default.

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:53>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Changes (by anonymous):

* cc: iv...@cs.washington.edu (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:54>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: assigned | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Comment (by dodgy...@gmail.com):

I am trying to store the output from a RSA encryption call in my
django database ... about 200 bytes I believe. I think BinaryField is
exactly what I am looking for. I applied the patch3 against r10175 and it
works.

Unfortunately it does appears to function a bit differently on sqlite
(local) v. mysql (remote).

On my local machine, the value of the field is as expected, but on the
remote machine, type(obj.cipher) gives me a <type 'buffer'> when
accessing the attribute which I don't know what to do with.

Is this the expected behaviour?

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:55>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody

Status: closed | Milestone:

Component: Database layer (models, ORM) | Version:

Resolution: fixed | Keywords:

Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Changes (by avk):

* status: assigned => closed
* resolution: => fixed

Comment:

Replying to [comment:55 dodgy...@gmail.com]:

> Is this the expected behaviour?

I agree that for a consistency reasons there should be always one type
returned, probably 'buffer',
since PostgreSQL developers mention that it is more efficient in this use
case. I will try to fix
it for all databases again so we will have consistent behavior.

Meanwhile, I would like to mention that from usage standpoint there is no
big difference between two types:
str can be converted to buffer, and buffer can be converted to 'str' when
required, using functions str(buffer)
or buffer(str).

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:56>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody

Status: reopened | Milestone:

Component: Database layer (models, ORM) | Version:

Resolution: | Keywords:

Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------
Changes (by avk):

* status: closed => reopened
* resolution: fixed =>

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:57>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:58>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Changes (by anonymous):

* cc: t.dj...@sandbox.cz (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:59>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Changes (by ruckc):

* cc: ru...@yahoo.com (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:60>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Changes (by clouserw):

* cc: clou...@gmail.com (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:61>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Changes (by sfllaw):

* cc: si...@akoha.com (added)

Comment:

Much like TextField uses LONGTEXT in MySQL, BlobField should use LONGBLOB.

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:62>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 0 |
---------------------------------------------------+------------------------

Changes (by bronger):

* cc: bro...@physik.rwth-aachen.de (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:63>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1

Needs_better_patch: 1 |
---------------------------------------------------+------------------------
Changes (by adamnelson):

* needs_better_patch: 0 => 1

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:64>

Django <http://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

--
You received this message because you are subscribed to the Google Groups "Django updates" group.
To post to this group, send email to django-...@googlegroups.com.
To unsubscribe from this group, send email to django-update...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-updates?hl=en.

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 1 |
---------------------------------------------------+------------------------

Changes (by anonymous):

* cc: pha...@rdrei.net (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:65>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 1 |
---------------------------------------------------+------------------------

Changes (by Odd_Bloke):

* cc: daniel....@credativ.co.uk (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:66>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)
---------------------------------------------------+------------------------
Reporter: sca...@nominum.com | Owner: nobody
Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:
Stage: Accepted | Has_patch: 1
Needs_docs: 0 | Needs_tests: 1
Needs_better_patch: 1 |
---------------------------------------------------+------------------------

Changes (by intgr):

* cc: intgr (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:67>

[Django]

#2417: Support for binary type fields (aka: bytea in postgres and VARBINARY in
mysql)

--------------------------------------------------------+-------------------
Reporter: scanner@… | Owner: nobody

Status: reopened | Milestone:
Component: Database layer (models, ORM) | Version:
Resolution: | Keywords:

Triage Stage: Accepted | Has patch: 1
Needs documentation: 0 | Needs tests: 1
Patch needs improvement: 1 |
--------------------------------------------------------+-------------------
Changes (by django@…):

* cc: django@… (added)

--
Ticket URL: <http://code.djangoproject.com/ticket/2417#comment:68>