deprecating ipaddressfield (at some time)

[Erik Romijn]

Hello all,

Since Django 1.4, we've added GenericIPAddressField, next to IPAddressField. The new GenericIPAddressField supports IPv4 as well as IPv6 addresses, and does normalisation of IPv6 addresses. It can also be configured to only accept IPv4 or IPv6 addresses.

As far as I know, IPAddressField has no current features that are not also available in a GenericIPAddressField. Therefore, I suggest that we, some time from now, deprecate IPAddressField, in favour of GenericIPAddressField.

For users, it is database-dependent whether IPAddressFields can just be replaced with GenericIPAddressFields: on PostgreSQL and SQLite, no changes are needed; schema changes are needed on MySQL and Oracle. Examples are listed in the 1.6 release notes[1], as we just made the same change for comments.

Is there anyone who sees any issues with this? I'm also not sure what timeline would be appropriate.

I have created ticket #20439[2] to keep track of this issue.

cheers,
Erik

[1] https://docs.djangoproject.com/en/dev/releases/1.6/#storage-of-ip-addresses-in-the-comments-app
[2] https://code.djangoproject.com/ticket/20439

[Erik Romijn]

Hello all,

On May 18, 2013, at 2:07 PM, Erik Romijn <er...@erik.io> wrote:
> As far as I know, IPAddressField has no current features that are not also available in a GenericIPAddressField. Therefore, I suggest that we, some time from now, deprecate IPAddressField, in favour of GenericIPAddressField.

Since this initial mail last May, I haven't seen any opinions against deprecating IPAddressField, on this list or in #20439 [1].

I therefore propose that we start the deprecation of IPAddressField in 1.7, in favour of GenericIPAddressField (which was introduced in 1.4). That means PendingDeprecationWarning in 1.7, DeprecationWarning in 1.8 and removal in 1.9. Obviously, I'll document how users can migrate - even if they don't use South it's quite simple.

What do you think?

[1] https://code.djangoproject.com/ticket/20439

cheers,
Erik

[Michael Manfre]

IPAddressField is meant for IPv4 addresses and GenericIPAddressField is for both IPv4 and IPv6. Most backends define different database data types for each of those fields. E.g. mysql is char(15) vs char(39). Forcing the larger data type on users doesn't make sense. 

IPAddressField and GenericIPAddressField are similar to the various integer fields. When BigIntegerField was added to the core, IntegerField and SmallIntegerField still had their purpose.

Regards,

Michael Manfre

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

[Florian Apolloner]

On Sunday, August 25, 2013 12:07:11 AM UTC+2, Michael Manfre wrote:

IPAddressField is meant for IPv4 addresses and GenericIPAddressField is for both IPv4 and IPv6. Most backends define different database data types for each of those fields. E.g. mysql is char(15) vs char(39). Forcing the larger data type on users doesn't make sense. 

 

It does make sense; storage is usually not such a concern and it's time that people start supporting IPv6; Even if we don't remove it now we should make the docs very clear about which field to use (basically you always want the generic variant). Also the normal IPAdressField has a few issues on postgres if it's nullable, so it's a nogo anyways if you want this to work over more databases… I don't see any strong reason why we shouldn't deprecate it.

[Erik Romijn]

Hi Michael,

On Aug 25, 2013, at 12:07 AM, Michael Manfre <mma...@gmail.com> wrote:

> IPAddressField is meant for IPv4 addresses and GenericIPAddressField is for both IPv4 and IPv6. Most backends define different database data types for each of those fields. E.g. mysql is char(15) vs char(39). Forcing the larger data type on users doesn't make sense.
>
> IPAddressField and GenericIPAddressField are similar to the various integer fields. When BigIntegerField was added to the core, IntegerField and SmallIntegerField still had their purpose.

I understand your comparison, but the situation is a little different here. The integer fields you refer to actually have different storage size requirements in many database backends. But for GenericIPAddressField in PostgreSQL and SQLite, the column types are the same - INET and TEXT.

On MySQL, the type is not char(15) vs char(39), but varchar(15) vs varchar(39). The storage size required for a varchar is the same for any length from 1 to 255: one byte to store the content length, and then the bytes of the content. So storage size or performance is not affected. On Oracle, it's VARCHAR2(15) vs VARCHAR2(39) - I assume they store it similar to MySQL.

In other words, if you replace an IPAddressField with a GenericIPAddressField and store the same data, storage size or database performance is not affected, even though the column type may need to be changed.

cheers,
Erik

[Aymeric Augustin]

On 18 mai 2013, at 14:07, Erik Romijn <er...@erik.io> wrote:

> As far as I know, IPAddressField has no current features that are not also available in a GenericIPAddressField. Therefore, I suggest that we, some time from now, deprecate IPAddressField, in favour of GenericIPAddressField.

Yes. Having both only creates confusion for users of Django.

--
Aymeric.

[Michael Manfre]

On Sun, Aug 25, 2013 at 6:13 AM, Erik Romijn <ero...@solidlinks.nl> wrote:

I understand your comparison, but the situation is a little different here. The integer fields you refer to actually have different storage size requirements in many database backends. But for GenericIPAddressField in PostgreSQL and SQLite, the column types are the same - INET and TEXT.

On MySQL, the type is not char(15) vs char(39), but varchar(15) vs varchar(39).

The code on master doesn't agree with your statement about mysql and sqlite.

sqlite - https://github.com/django/django/blob/master/django/db/backends/sqlite3/creation.py#L26

mysql - https://github.com/django/django/blob/master/django/db/backends/mysql/creation.py#L23

 

The storage size required for a varchar is the same for any length from 1 to 255: one byte to store the content length, and then the bytes of the content. So storage size or performance is not affected. On Oracle, it's VARCHAR2(15) vs VARCHAR2(39) - I assume they store it similar to MySQL.

I don't think that is true for all databases, nor should Django make that type of assumption. Unless I've misunderstood how MSSQL works, which is possible, it only adds a few bytes of overhead to the actual data.

 

In other words, if you replace an IPAddressField with a GenericIPAddressField and store the same data, storage size or database performance is not affected, even though the column type may need to be changed.

This is probably true for some databases, but not guaranteed to be true for all.

Regards,

Michael Manfre

[Erik Romijn]

Hi Michael,

On Aug 25, 2013, at 6:27 PM, Michael Manfre <mma...@gmail.com> wrote:
> The code on master doesn't agree with your statement about mysql and sqlite.
> sqlite - https://github.com/django/django/blob/master/django/db/backends/sqlite3/creation.py#L26

SQLite doesn't actually have a CHAR type - it will consider both of these columns to be the TEXT type: http://www.sqlite.org/datatype3.html

> mysql - https://github.com/django/django/blob/master/django/db/backends/mysql/creation.py#L23

You are correct - I seem to have remembered incorrectly.

>> The storage size required for a varchar is the same for any length from 1 to 255: one byte to store the content length, and then the bytes of the content. So storage size or performance is not affected. On Oracle, it's VARCHAR2(15) vs VARCHAR2(39) - I assume they store it similar to MySQL.
>>
> I don't think that is true for all databases, nor should Django make that type of assumption. Unless I've misunderstood how MSSQL works, which is possible, it only adds a few bytes of overhead to the actual data.

MSSQL is not a supported database in Django itself, so I have no idea, or control over, how the (Generic)IPAddressField types are stored in there. Let alone what the differences in storage size might be.

>> In other words, if you replace an IPAddressField with a GenericIPAddressField and store the same data, storage size or database performance is not affected, even though the column type may need to be changed.
>>
> This is probably true for some databases, but not guaranteed to be true for all.

With your correction on the MySQL datatypes, this is indeed not true for MySQL. It would introduce an overhead of 24 bytes. However, countering that are several good reasons to proceed with deprecation.

IPAddressField has no model validation, which means that it will often allow storing IPv6 addresses. This may work, or may result in silent truncation or an error, depending on the database backend and it's settings. Should storing the IPv6 address work, then trying to edit that object through a ModelForm later, e.g. in the admin, will fail. These limitations are in no way obvious to any IPAddressField user.

IPv6 addresses are not just encountered by users that consciously make a decision to support IPv6, and therefore would look deeper into support in fields. It is fairly common to see IPv6-mapped IPv4 addresses, like ::ffff::192.0.2.1, in REMOTE_ADDR. IPAddressField may truncate these silently or fail the query.

As Aymeric points out, it is just confusing to our users. IPAddressField also has some issues regarding the handling of blank strings vs None. This may result in unexpected behaviour or failures on some databases, but can't be fixed as it would break backwards compatibility. These are fixed in GenericIPAddressField.

To sum it up, the overhead may be an issue for users for which storage is very critical, and who will only need to store IPv4 addresses. Particularly the latter situation seems doubtful to me, as IPv6 adoption will only increase. However, even if these users exist, they can still create their own field. But users like this will be extremely rare, compared to those being hit by issues or confusion caused by the continued existence of IPAddressField.

cheers,
Erik

[Marc Tamlyn]

+1 to deprecating the old field. This caused me some confusion. Now migrations are in master, the upgrade path will be much simpler.

Is the legacy field releasable as a 3rd party package? It seems to be this would be difficult due to differing behaviour across DBs.

Marc

[Aymeric Augustin]

On 25 août 2013, at 19:14, Erik Romijn <ero...@solidlinks.nl> wrote:

> To sum it up, the overhead may be an issue for users for which storage is very critical, and who will only need to store IPv4 addresses.

Users with such stringent storage requirements would be better off storing IPv4 addresses as 32 bits integers. Using 15 characters takes 3,75 times more space than necessary, after all :)

> Particularly the latter situation seems doubtful to me, as IPv6 adoption will only increase. However, even if these users exist, they can still create their own field. But users like this will be extremely rare, compared to those being hit by issues or confusion caused by the continued existence of IPAddressField.

Reducing the risk for data loss or crashing bugs is well worth a small waste of space for the few people who are guaranteed never to encounter an IPv6 address — that is, people whose software run on private IPv4 networks disconnected from the Internet that will never be upgraded to IPv6.

--
Aymeric.

[Some Developer]

I agree. I don't see any need at all to keep both around. IPv4 addresses
have already been depleted in two areas and the US is due to follow
shortly. Soon it will be IPv6 or nothing for new ventures unless they
can get hold of their IPv4 addresses from someone else.