Making max_length argument optional

1,669 views
Skip to first unread message

Podrigal, Aron

unread,
Jul 29, 2015, 12:27:59 AM7/29/15
to Django developers (Contributions to Django itself)

Hi,

I am using postgresql and I prefer my VARCHAR columns not to have a length limit. Is there any particular reason why max_length arg to fields is required. If for  compatibility with other database backends we can have some sane default if it is None.

Loïc Bistuer

unread,
Jul 29, 2015, 1:41:01 AM7/29/15
to django-d...@googlegroups.com
Hi Aron,

I'm +1 on addressing this, I often don't care about max_length, but I still want a terse form representation. In my projects I use a subclass of TextField that sets a TextInput wiget in its formfield() method. But that's not very elegant: it requires a custom field for a common use-case, and it's tightly coupled to Django forms (i.e. won't work with other form libraries and non-HTML forms).

Now how we address this depends on the meaning we want to give to models fields like CharField or TextField, more specifically whether the emphasis is on end user's perspective or on mapping 1:1 to database concepts.

If we see CharField and TextField as respectively "short/single line field" and "long/multiline field" from a user's perspective, then it makes sense to lift the max_length requirement and use `db_type='TextField'` when max_length is None.

However if we want to stay close to their database equivalents, then we could have a display hint on TextField (e.g. TextField(display_multiline=False)) and Field.formfield() would use that hint to provide a TextInput widget instead of a Textarea.

Personally I'd much prefer option 1, it's significantly less verbose and more often than not I don't want a length constraint, especially considering that there is no performance penalty in doing so with all the databases that I care about. Also considering we want first class virtual field support (and plan on turning existing concrete fields into virtual fields) I don't think it's a big problem to break from the existing models.Field/db types mapping.

Cheers,
Loïc

> On Jul 29, 2015, at 11:27, Podrigal, Aron <ar...@guaranteedplus.com> wrote:
>
> Hi,
>
> I am using postgresql and I prefer my VARCHAR columns not to have a length limit. Is there any particular reason why max_length arg to fields is required. If for compatibility with other database backends we can have some sane default if it is None.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yjwzpk%2BN2%3D4WWbmj5zu2Gwu67G0Sme1f8xQa%2BKrzSQ2Rw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

Tim Graham

unread,
Jul 29, 2015, 9:28:50 AM7/29/15
to Django developers (Contributions to Django itself), loic.b...@gmail.com

Podrigal, Aron

unread,
Jul 29, 2015, 12:25:34 PM7/29/15
to Django developers (Contributions to Django itself), loic.b...@gmail.com
I see models as what defines the database ddl and and not a representation from a  end users perspective. And I see the tight coupling between the 2 improper. Although in most cases the assumptions of representation and mapping of the 2 is mostly accepted, explicit is better than implicit. Of course we can have defaults that would work for most cases, but we need a way to override that. For me I rarely use the builtin generated widgets, I use restframework so all I'm interested in a model is for db definition in a highly customizable way.

I would opt for a db_length kwarg which can in turn be used for any field type eg. Timestamps


For more options, visit https://groups.google.com/d/optout.



--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110'   '1010000', '1101111', '1100100', '1110010', '1101001', '1100111', '1100001', '1101100'

P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'

Aymeric Augustin

unread,
Jul 30, 2015, 5:17:56 AM7/30/15
to django-d...@googlegroups.com, loic.b...@gmail.com
> Le 29 juil. 2015 à 18:25, Podrigal, Aron <ar...@guaranteedplus.com> a écrit :
>
> I see models as what defines the database ddl and and not a representation from a end users perspective.

Django models do a bit more than this: see the `validators` argument, EmailField, etc.
> And I see the tight coupling between the 2 improper.

I understand your perspective; it's a trade-off between purity and convenience and the optimal answer depends on each project.
> For me I rarely use the builtin generated widgets, I use restframework so all I'm interested in a model is for db definition in a highly customizable way.

Please make sure your proposals also account for the various way Django can be and has been used :-)

--
Aymeric.

Podrigal, Aron

unread,
Sep 21, 2015, 12:15:16 AM9/21/15
to Django developers (Contributions to Django itself), loic.b...@gmail.com
Iv'e started to work on this ticket [1] today and here [2] is my work so far. I'd appreciate some feedback on the implementation.
There are a couple ways to handle the database type length. 

1) Separate types one with a length spec and one without (as mentioned by Ben Davis in the ticket)
2) Some interface to get the type length specs (the approach I took)

The second option more generic, and provides an interface for any type to specify a length or scale precision.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

For more options, visit https://groups.google.com/d/optout.

Paulo Maciel

unread,
Sep 21, 2015, 12:21:57 PM9/21/15
to Django developers (Contributions to Django itself)
+1 max_length optional

Tom Evans

unread,
Sep 21, 2015, 12:54:55 PM9/21/15
to django-d...@googlegroups.com
On Mon, Sep 21, 2015 at 5:21 PM, Paulo Maciel
<paulosou...@gmail.com> wrote:
> +1 max_length optional
>

I'm slightly worried from a DB point of view. Whilst blessed
PostgreSQL supports variable length fields seemingly without issues,
other DB engines (MySQL, for instance), have significant performance
issues using a TEXT field versus a VARCHAR field - particularly on
indexed fields, it must do a row read to retrieve data, rather than
use an indexed value.

Remember that VARCHAR columns can be fully indexed, whilst TEXT
columns can only have the first N (specified by user) characters
indexed, and reading the data will always require a row read, even if
the text is shorter than the indexed size.

In the ticket, it states that

"""
If I read the oracle docs correctly, specifying a length is mandatory
for VARCHAR2, and the maximum is 4000 bytes. For MySQL, it's 65K
bytes, for PostgreSQL and SQLite it's very large.
"""

For MySQL, it is 64k (65,535 bytes is *not* 65k!), and that is not the
whole story - 64k is the maximum record size - all your columns must
fit within that limit, not just one column in the record.

http://dev.mysql.com/doc/refman/5.7/en/column-count-limit.html

I'm slightly concerned as a MySQL/MariaDB user that this will lead to
"reusable" apps that are not performant on MySQL. Given that we
(generally) let Django define the database tables based upon models,
it seems much more important that the definition is correct and high
performing in all supported backends than to avoid choosing what
database columns you require in the tables - after all, that is what
the TextField is for, to allow the programmer to specify "this is an
arbitrary length text field".

I'm all for DB engines that support arbitrary length varchar fields to
use that when you specify a TextField. I'm very not keen on having
MySQL represent the majority of char fields as TEXT because no-one
could be bothered to specify the appropriate length.

Cheers

Tom

Christophe Pettus

unread,
Sep 21, 2015, 2:46:42 PM9/21/15
to django-d...@googlegroups.com

On Sep 21, 2015, at 9:54 AM, 'Tom Evans' via Django developers (Contributions to Django itself) <django-d...@googlegroups.com> wrote:
> I'm slightly worried from a DB point of view.

I have to agree, even speaking as PostgreSQL geek. While VARCHAR and TEXT are implemented the same way in PostgreSQL, conceptually they're different things. I don't think the relatively modest benefit of having no default justifies the problems that result on other platforms.

--
-- Christophe Pettus
x...@thebuild.com

Podrigal, Aron

unread,
Sep 21, 2015, 5:49:44 PM9/21/15
to Django developers (Contributions to Django itself)
We're not talking about representing all CharFields as TEXT, it is about choosing a sane length as the default for the varchar datatype. So if you would not specify max_length, for MySQL it would be varchar(255), for oracle it would be varchar(4000 / bytes_per_char_for NLS_CHARACTERSET) and for PostgreSQL it would be just VARCHAR without a length.

I would like some comments / help for how to compute the size of the string in bytes for Oracle. unless we can opt for using a default length of 255 for all database backends besides postgres.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

For more options, visit https://groups.google.com/d/optout.

Christophe Pettus

unread,
Sep 21, 2015, 8:42:40 PM9/21/15
to django-d...@googlegroups.com

On Sep 21, 2015, at 2:49 PM, "Podrigal, Aron" <ar...@guaranteedplus.com> wrote:

> We're not talking about representing all CharFields as TEXT, it is about choosing a sane length as the default for the varchar datatype.

But that means notably different schemas on different backends, for not an obvious gain. What's the benefit there?

Podrigal, Aron

unread,
Sep 21, 2015, 8:49:11 PM9/21/15
to Django developers (Contributions to Django itself)
Different schemas?? Schema will always be different for each database backend according to its datatypes. I really don't understand what your concern is. In any case your free to specify a max_length=N where it will be the same for all backends.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

For more options, visit https://groups.google.com/d/optout.

Christophe Pettus

unread,
Sep 21, 2015, 8:52:39 PM9/21/15
to django-d...@googlegroups.com

On Sep 21, 2015, at 5:49 PM, "Podrigal, Aron" <ar...@guaranteedplus.com> wrote:

> Different schemas?? Schema will always be different for each database backend according to its datatypes.

It means if you specify a CharField without a length, you don't know how many characters it can accept without error. That doesn't seem like something we should make a change to accept.

> I really don't understand what your concern is.

The current behavior seems entirely reasonable, and I'm not sure I understand what problems it is causing. Specifying a maximum length on a CharField is not just a random habit; it should be done as a way of sanity checking the value to a reasonable length. Sometimes, that's natural to the data (there are no 50 character telephone numbers or 5000 character email addresses), sometimes it's just a way of making sure that something bad doesn't get into the database.

Podrigal, Aron

unread,
Sep 21, 2015, 9:12:25 PM9/21/15
to Django developers (Contributions to Django itself)
The reason for having a max_length set to None, is because that's what I want for my database columns to be in Postgres, and for MySQL I don't care about the length too, I always choose varchar(255) just for because it is required for the database backend. And for validation of the max length, I would argue that just like validating the max I want to validate the min. In short, I want to have more control on my database schema.

And for your concern, there will be a MaxLengthValidator added to the validators to validate the users input does not exceed the database backends maximum length just like when you set max_length explicitly. Also you can manually provide a validator if your user input validation rule is to be less than the db maximum length. See [1]. The MaxLengthValidator should actually only be added if there isn't yet any MaxLengthValidator explicitly given.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

For more options, visit https://groups.google.com/d/optout.

Christophe Pettus

unread,
Sep 21, 2015, 10:03:18 PM9/21/15
to django-d...@googlegroups.com

On Sep 21, 2015, at 6:12 PM, "Podrigal, Aron" <ar...@guaranteedplus.com> wrote:

> The reason for having a max_length set to None, is because that's what I want for my database columns to be in Postgres, and for MySQL I don't care about the length too, I always choose varchar(255) just for because it is required for the database backend.

Well, that's not a practice I think we need to go to great lengths to support. If you *really* *must* have a VARCHAR field without a length, you can always use a migration to strip it off.

Podrigal, Aron

unread,
Sep 21, 2015, 10:10:48 PM9/21/15
to Django developers (Contributions to Django itself)
There is actually another reason to not have to specify a max_length which was mentioned earlier, is because most of the time you don't care about that and is just tedious to have to specify that when you can get it to work without it. Default values has always been here for that reason.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

For more options, visit https://groups.google.com/d/optout.

Christophe Pettus

unread,
Sep 21, 2015, 10:16:29 PM9/21/15
to django-d...@googlegroups.com

On Sep 21, 2015, at 7:10 PM, "Podrigal, Aron" <ar...@guaranteedplus.com> wrote:

> There is actually another reason to not have to specify a max_length which was mentioned earlier, is because most of the time you don't care about that and is just tedious to have to specify that when you can get it to work without it. Default values has always been here for that reason.

I'm afraid I must disagree that "most of the time you don't care about it." I certainly do. I'm always reasonably careful to specify a max_length that corresponds to the underlying data.

There's no "sensible" default for max_length. It entirely depends on the data you are storing. Picking a backend-specific max_length means that application writers now have no idea how much data the CharField can store: 1GB-ish-depending-on-encoding? 255? 4000 or something less depending on Oracle's encoding?

Requiring a max_length is enforcing a good practice.

Shai Berger

unread,
Sep 21, 2015, 10:22:28 PM9/21/15
to django-d...@googlegroups.com
Hi,

I just went over the ticket and thread. There seem to be two concerns mixed
here.

Most people who expressed support for the feature have just been annoyed at
the need to specify a length when all they want to say is "a reasonably short
string" or "a one-liner".

Aron, on the other hand, seems to aim mostly at an option to remove the length
limitation completely.

I can see why "unlimited" makes sense, sometimes, on Postgres. But I think
it's a bad default for general use, especially in reusable apps.

I also don't like the interpretation of "unlimited" as "backend-specified max",
basically for the reasons Christoph mentioned (same schema on different
backends), and also because I'm concerned about the performance implications
of unnecessarily large fields and about the ambiguity of introspection.

I'd solve the "need to specify" issue by setting a default that is
intentionally smaller than the smallest (core) backend limitation, say 128.
I"d make an "unlimited length text field" a new type of field, explicitly not
supported on MySql and Oracle; and I'd suggest that it can live outside core
for a while. so we may get an impression of how popular it really is.

My 2 cents,
Shai.

Christophe Pettus

unread,
Sep 21, 2015, 10:26:35 PM9/21/15
to django-d...@googlegroups.com

On Sep 21, 2015, at 7:22 PM, Shai Berger <sh...@platonix.com> wrote:

> I'd solve the "need to specify" issue by setting a default that is
> intentionally smaller than the smallest (core) backend limitation, say 128.

I'd be OK with that. Not wild, because I think that having to specify max_length is good discipline, but not everyone likes oatmeal, either. :)

> I"d make an "unlimited length text field" a new type of field, explicitly not
> supported on MySql and Oracle; and I'd suggest that it can live outside core
> for a while. so we may get an impression of how popular it really is.

We kind of have that: TextField. The problem is that TextField has very different performance characteristics and implementation details on PostgreSQL vs MySQL and Oracle. I don't think we need another: If you know you are running on PostgreSQL, you just use TextField, and if you are either targeting a different database, or writing one that runs on multiple ones, you probably want CharField with a specific length.

Podrigal, Aron

unread,
Sep 21, 2015, 10:33:49 PM9/21/15
to Django developers (Contributions to Django itself)
On Mon, Sep 21, 2015 at 10:26 PM, Christophe Pettus <x...@thebuild.com> wrote:

On Sep 21, 2015, at 7:22 PM, Shai Berger <sh...@platonix.com> wrote:

> I'd solve the "need to specify" issue by setting a default that is
> intentionally smaller than the smallest (core) backend limitation, say 128.

I'd be OK with that.  Not wild, because I think that having to specify max_length is good discipline, but not everyone likes oatmeal, either. :)

OK, this satisfies me too. 

> I"d make an "unlimited length text field" a new type of field, explicitly not
> supported on MySql and Oracle; and I'd suggest that it can live outside core
> for a while. so we may get an impression of how popular it really is.

We kind of have that: TextField.  The problem is that TextField has very different performance characteristics and implementation details on PostgreSQL vs MySQL and Oracle.  I don't think we need another: If you know you are running on PostgreSQL, you just use TextField, and if you are either targeting a different database, or writing one that runs on multiple ones, you probably want CharField with a specific length.

While this makes sense, the use of TextField is also for the purpose of having rendered a text widget for ModelForms. So we should allow a max_length of None for both.
 

--
-- Christophe Pettus
   x...@thebuild.com

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

For more options, visit https://groups.google.com/d/optout.

Josh Smeaton

unread,
Sep 22, 2015, 12:22:33 AM9/22/15
to Django developers (Contributions to Django itself)
> While this makes sense, the use of TextField is also for the purpose of having rendered a text widget for ModelForms. So we should allow a max_length of None for both.

If you use Oracle (or mysql I guess), you get used to manually changing the form field widget to be a TextArea if that's what you want for your UI. Using TextField with Oracle is a horrible experience unless you must have it.

Aymeric Augustin

unread,
Sep 22, 2015, 1:49:44 AM9/22/15
to django-d...@googlegroups.com
Hi Aron,

On 22 sept. 2015, at 03:12, Podrigal, Aron <ar...@guaranteedplus.com> wrote:

> And for your concern, there will be a MaxLengthValidator added to the validators to validate the users input does not exceed the database backends maximum length just like when you set max_length explicitly.

This isn’t possible in a project that uses multiple databases, say PostgreSQL and Oracle. The form layer cannot say which length is correct because it has doesn’t know in what database the data will be saved.

--
Aymeric.




Aymeric Augustin

unread,
Sep 22, 2015, 1:56:51 AM9/22/15
to django-d...@googlegroups.com
Hi Shai,

On 22 sept. 2015, at 04:22, Shai Berger <sh...@platonix.com> wrote:

> I'd solve the "need to specify" issue by setting a default that is
> intentionally smaller than the smallest (core) backend limitation, say 128.

I would pick the highest value supported by all core backends (probably 255
for MySQL, unless there’s something about indexes and multi-byte encodings
that I forget) in order to minimize the need to increase it.

If we go for a lower value, I suggest to pick something totally arbitrary like
100 to make it clear that it isn't a technical limitation.

> I"d make an "unlimited length text field" a new type of field, explicitly not
> supported on MySql and Oracle; and I'd suggest that it can live outside core
> for a while. so we may get an impression of how popular it really is.

The main use case seems to be “VARCHAR() on PostgreSQL”. What about defining a
slight variant of CharField in django.contrib.postgres that merely makes the
max_length argument default to None?

--
Aymeric.

Podrigal, Aron

unread,
Sep 22, 2015, 1:58:05 AM9/22/15
to Django developers (Contributions to Django itself)

Aymeric, thanks for clarification, my bad I missed that one. We will be using max_length=255 or 128 for the default as Shia proposed.

BTW how is this handled by integer_ranges on per database backend? Also, using checks_framework can we validate this setting?

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

Aymeric Augustin

unread,
Sep 22, 2015, 2:03:53 AM9/22/15
to django-d...@googlegroups.com
On 22 sept. 2015, at 07:57, Podrigal, Aron <ar...@guaranteedplus.com> wrote:

We will be using max_length=255 or 128 for the default as Shia proposed.

Would you mind giving a few hours for other contributors to react? I was asleep while you were having this discussion; not every contributor is hooked to django-developers, especially technical board members.

BTW how is this handled by integer_ranges on per database backend?

I was asking myself the same question after seeing https://code.djangoproject.com/ticket/14094#comment:15. I don’t know; perhaps it doesn’t! It’s worth investigating.

-- 
Aymeric.




Podrigal, Aron

unread,
Sep 22, 2015, 2:56:58 AM9/22/15
to Django developers (Contributions to Django itself)

Ok, I'll wait for other contributors to react.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

Remco Gerlich

unread,
Sep 22, 2015, 7:56:04 AM9/22/15
to django-d...@googlegroups.com


On Mon, Sep 21, 2015 at 11:49 PM, Podrigal, Aron <ar...@guaranteedplus.com> wrote:
We're not talking about representing all CharFields as TEXT, it is about choosing a sane length as the default for the varchar datatype. So if you would not specify max_length, for MySQL it would be varchar(255), for oracle it would be varchar(4000 / bytes_per_char_for NLS_CHARACTERSET) and for PostgreSQL it would be just VARCHAR without a length.

I feel that IF a "sane length as the default" exists, then it must be the same for all database backends. The errors you get from a ModelForm on a web page shouldn't change depending on the database backend!

Maybe django.contrib.postgres could have a ArbitraryLengthCharField?

Greetings,
Remco Gerlich

Tim Chase

unread,
Sep 22, 2015, 7:59:45 AM9/22/15
to Aymeric Augustin, django-d...@googlegroups.com
On 2015-09-22 07:49, Aymeric Augustin wrote:
> > And for your concern, there will be a MaxLengthValidator added to
> > the validators to validate the users input does not exceed the
> > database backends maximum length just like when you set
> > max_length explicitly.
>
> This isn’t possible in a project that uses multiple databases, say
> PostgreSQL and Oracle. The form layer cannot say which length is
> correct because it has doesn’t know in what database the data will
> be saved.

Could this be resolved with allowing max_length=None (or some other
atom such as just object() that would have a unique id() that could
be tested with "is") to specify that the back-end uses the max-allowed
value? For Postgres, this would then use VARCHAR() with no limit,
while on MySQL, that could be 255 or whatever.

One could then include a helper function that would use the current
settings to check a string (such as from a form) to ensure that it
doesn't exceed the field's database-specific max-length (or whatever
the specified max-length value is).

-tim




Collin Anderson

unread,
Sep 22, 2015, 8:41:10 AM9/22/15
to django-d...@googlegroups.com
If anyone's curious, the mysql situation is as crazy as you might expect. :)

The max is only determined only by the total row size (65,535 bytes) and the index size (767 bytes default).

Mysql defaults to only allowing 3-byte (no emoji) unicode characters, so 65,535/3=21,845 max across the entire row (the sum of all of the maxes of all char/varchar columns), and each indexed field only gets 767/3=255 characters by default.

If you change to 4-byte unicode characters, (which you should, but django doesn't really help you out #18392), your max_lengths can add up to 65,535/4=16,383 characters, and if you want the field to be indexed, you only get 191 characters (using the default index size). It's possible to only index the first 767/4=191 characters of the field, but django doesn't really support that.

Basically, 255 works pretty well by default, allowing 65,535/3/255=85 3-byte fields per row, and indexes just work if you stick to the default settings.



--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

Michael Manfre

unread,
Sep 22, 2015, 10:01:23 AM9/22/15
to django-d...@googlegroups.com
I'm -1 on having a default max_length on CharField. We should encourage developers to be more explicit when defining their models, not less. I've inherited way too many databases that suffered from "VARCHAR(255)" being chosen as the arbitrary "any field storing even a few characters will be this length".

Instead of adding a default (and encouraging bad database schemas), what are thoughts about adding a helper function to allow people to define their own new CharField as a one liner? E.g. ShortCharField = make_max_length_char_field(255)

I agree that the Postgresql specific field should exist in django.contrib.postgres.

Regards,
Michael Manfre


For more options, visit https://groups.google.com/d/optout.



--
GPG Fingerprint: 74DE D158 BAD0 EDF8

Tom Evans

unread,
Sep 22, 2015, 10:05:55 AM9/22/15
to django-d...@googlegroups.com
On Tue, Sep 22, 2015 at 1:49 AM, Podrigal, Aron
<ar...@guaranteedplus.com> wrote:
> Different schemas?? Schema will always be different for each database
> backend according to its datatypes. I really don't understand what your
> concern is. In any case your free to specify a max_length=N where it will be
> the same for all backends.

This change would allow a django site running on postgres that you
could not dump the data from and reimport in to mysql without data
loss. It would make it easy to write non-portable apps without
thinking.

>
> While this makes sense, the use of TextField is also for the purpose of
> having rendered a text widget for ModelForms. So we should allow a
> max_length of None for both.
>

(I'm replying to two emails from you in one, hope that is alright)

No, TextField is used to designate an arbitrary length text field.
Having a widget.Textarea as the default input for this field is just a
default, it is perfectly correct to override the defaults either for
CharField or TextField to provide the desired widget type.

One should not be using models.TextField because a widget.Textarea is
desired for forms, instead it is to tell the database backend that the
column for this field should be an arbitrary length text type.

If the problem that is to be solved is that there is no easy way to
specify an arbitrary length text field that uses an input, this can be
solved with a very simple class:

class TextFieldWithInput(models.TextField):
def formfield(self, **kwargs):
defaults = {'widget': forms.TextInput}
defaults.update(kwargs)
return super(TextfieldWithInput, self).formfield(**defaults)

The main argument seems to be "Its a pain to think about the size of
my data when defining my database tables"; if we aren't thinking about
it then, then when do we think about it? Its kind of important..

Cheers

Tom

Christophe Pettus

unread,
Sep 22, 2015, 1:14:03 PM9/22/15
to django-d...@googlegroups.com

On Sep 22, 2015, at 1:01 AM, Remco Gerlich <re...@gerlich.nl> wrote:

> Maybe django.contrib.postgres could have a ArbitraryLengthCharField?

Just a note that, on PostgreSQL, that's exactly what TextField is. There might be a use for a field that creates a VARCHAR without length on PostgreSQL, but I can't think of it.

Tom Christie

unread,
Sep 23, 2015, 1:02:20 PM9/23/15
to Django developers (Contributions to Django itself)
I'm with Tom here.
Forcing `max_length` to always be set on CharField feels like the right decision. Having a default there seems unnecessary obscure, and more likely to lead to untested/unnoticed failure cases.
It *could* be that we'd allow `max_length=None` to explicitly turn off the validation, but I'm not sure if that's better than forcing developers who want unvalidated lengths to use TextField.

Wim Feijen

unread,
Feb 27, 2016, 7:55:23 AM2/27/16
to Django developers (Contributions to Django itself)
Hi guys,

I'd like to reach consensus about https://code.djangoproject.com/ticket/14094

In the discussion, I saw three options:

1. Adding a default max_length = None, which may in practice differ per database backend.
2. Adding a default max_length of a value which can be used across databases. Lengths which have been mentioned are: 100, 128, 191 and 255 characters and 4000 bytes.
3. Keep everything as is, being as explicit as possible.

Option 1 does not allow for apps and data to be reused across several apps and several core developers do not support this idea.

I'm in favour of option 2, because it makes for shorter code and more importantly, it helps programmers choose a value which actually can be reused across multiple databases (I didn't know about the possible limit of 191 characters on mysql! Did you?).

I dislike how it is now (option 3), and I don't think it is beneficial here to force people to be explicit, because people are now using arbitrary values anyway. 255 is commonly used, the poll app of the Django tutorial uses 200. Actually I believe it is a big mess in practice, I see defaults of 100, 200, 255, 300, 1000 seemingly randomly used. For example, try searching for CharField in your favourite apps or CMSes.

In addition, adding a default for max_length still allows to be explicit when it matters. And it makes life simple when it doesn't matter, and simple is better than complex.

What would be a good default then? I am in favour of making it as big as possible while still working on all databases, which would either be 191 of 255 in order to support mysql.

Wim

Florian Apolloner

unread,
Feb 27, 2016, 8:01:52 AM2/27/16
to Django developers (Contributions to Django itself)
-1 on adding an arbitrary default like 100/128… If people choose a CharField the should set max_length manually. In that sense I am for option 3.

Cheers,
Florian

Wim Feijen

unread,
Feb 27, 2016, 8:33:27 AM2/27/16
to Django developers (Contributions to Django itself)
Hi Florian,

Can you please elaborate why the current situation is better? In practice, most people add max_length=255 everytime by hand. I value your opinion but I fail to understand your reasoning. Could you please explain further (if you have time)?  

Wim 

Florian Apolloner

unread,
Feb 27, 2016, 9:56:26 AM2/27/16
to Django developers (Contributions to Django itself)
Hi Wim,


On Saturday, February 27, 2016 at 2:33:27 PM UTC+1, Wim Feijen wrote:
Can you please elaborate why the current situation is better?

I wouldn't say it is better, just equally good/bad ;)
 
In practice, most people add max_length=255 everytime by hand. I value your opinion but I fail to understand your reasoning. Could you please explain further (if you have time)?  

In practice, that would mean most people do not care about the length, so they should be using a TextField, no? (See next paragraph for even another option) If you want something which works on multiple databases, 255 is a nonsensical default anyways -- with all the love for emojis out there, you'd be limiting yourself to 191 (thank MySQL for that 767/4 = 191). What worries me even more, is that changing this "default" (to accommodate for a new database for instance) will require database migrations, so all of a sudden upgrading django will result in migrations for all of your apps.

That all said, if it were for me I'd deprecate CharField and only support TextField and give it an optional max_length (and a way to switch between TextArea and LineEdit for UI) -- but that is probably just because on Postgres there is no fundamental difference between them. From an enduser perspective, it is all text, the only difference is a constraint in length, which does not explain the naming of this fields very well anyways (unless you are coming from a SQL background where you might be used to that distinction). As for the UI issues: A TextField with an optional max_length could use a <input> as default if max_length is specified and <textarea> otherwise (with a boolean flag to switch that too). Sadly, if we switch from max_length=None to max_length=xxx, this would mean an alteration of the database type on most databases, I am not sure if Django can handle that currently.

I hope this clears it up a little bit,
Florian

Shai Berger

unread,
Feb 27, 2016, 10:44:18 AM2/27/16
to django-d...@googlegroups.com
Hi Florian, Wim and all,

On Saturday 27 February 2016 16:56:25 Florian Apolloner wrote:
> Hi Wim,
>
> On Saturday, February 27, 2016 at 2:33:27 PM UTC+1, Wim Feijen wrote:
> > Can you please elaborate why the current situation is better?
>
> I wouldn't say it is better, just equally good/bad ;)
>

I disagree.

> > In practice, most people add max_length=255 everytime by hand. I value
> > your opinion but I fail to understand your reasoning. Could you please
> > explain further (if you have time)?
>
> In practice, that would mean most people do not care about the length, so
> they should be using a TextField, no?

Not really (see below).

> (See next paragraph for even another
> option) If you want something which works on multiple databases, 255 is a
> nonsensical default anyways -- with all the love for emojis out there,
> you'd be limiting yourself to 191 (thank MySQL for that 767/4 = 191). What
> worries me even more, is that changing this "default" (to accommodate for a
> new database for instance) will require database migrations, so all of a
> sudden upgrading django will result in migrations for all of your apps.
>

I think this is why we should be a little more cautious and go for 128 or 100,
rather than "the current maximum all-backend-compatible value".

> That all said, if it were for me I'd deprecate CharField and only support
> TextField and give it an optional max_length (and a way to switch between
> TextArea and LineEdit for UI) -- but that is probably just because on
> Postgres there is no fundamental difference between them. From an enduser
> perspective, it is all text, the only difference is a constraint in length,

No, that's not the only difference. There's also an understanding that
CharField's are one-liners, that is, there is an implicit limitation (usually
only enforced by UI) that newlines are not allowed in the text.

And that makes it a different kind of beast, in general. For example, I don't
think there are many CharField's whose contents are interpreted as markup. On
the other hand, you don't see many TextField's with Regex validators. In a
user perspective, these are really different data types. I find your suggestion
analogous to a suggestion to remove IntegerField, because "it's just a
FloatField with a constraint on scale".

My 2 cents,
Shai.

Florian Apolloner

unread,
Feb 27, 2016, 4:57:19 PM2/27/16
to Django developers (Contributions to Django itself)
Hi Shai,


On Saturday, February 27, 2016 at 4:44:18 PM UTC+1, Shai Berger wrote:
I think this is why we should be a little more cautious and go for 128 or 100,
rather than "the current maximum all-backend-compatible value".

Which somewhat speaks for my point, cause we do not know if the next backend will need less than 100 chars (But I do agree that this is not likely). Tough I think that 100 (or 191 for matter) is a ridiculously low limit. In that sense, allowing max_length=None is probably the safest solution -- even though we do not know when database backends will crash on this, but we do not know this for text fields either currently.

> That all said, if it were for me I'd deprecate CharField and only support
> TextField and give it an optional max_length (and a way to switch between
> TextArea and LineEdit for UI) -- but that is probably just because on
> Postgres there is no fundamental difference between them. From an enduser
> perspective, it is all text, the only difference is a constraint in length,

No, that's not the only difference. There's also an understanding that
CharField's are one-liners, that is, there is an implicit limitation (usually
only enforced by UI) that newlines are not allowed in the text.

This is a leaky abstraction at best in my opinion. The "understanding" is mostly coined by the documentation in the sense that it mentions <textarea> for TextField and <input> for CharField. So it is kind of illogical to not allow a max_length on TextField. Even with model.full_clean there is no restriction aside from the length, in that sense a CharField behaves completely the same as TextField (database issues with index lengths etc excluded).

For example, I don't
think there are many CharField's whose contents are interpreted as markup.

Maybe, maybe not, I do not see how this makes a difference here though. (Same goes for Regex validation).
 
In a user perspective, these are really different data types.

Which can be dangerous on it's own, I'd have to check, but I am pretty sure that nothing prevents the user from submitting newlines in a CharField.
 
I find your suggestion
analogous to a suggestion to remove IntegerField, because "it's just a
FloatField with a constraint on scale".

I'd suggest a DecimalField cause I do not like FloatFields either, but yeah ;)

Do not get me wrong, I somewhat agree with everything you say (in the end I think it is just a matter of taste) -- but I also think that explicit is better than implicit. From a security point of view, I agree that length validation should be done whenever it makes sense [which is basically always, there is no point in allowing to post the maximum length of a TextField, especially not if it contains markup which you have to render again ;)], so a CharField with a default max_length seems like a good idea. On the other hand I think that it will make the distinction for new users even harder. Assume we change the CharField to have a default max_length, the first (and probably only thing) users will notice is that CharField is single line. I am somewhat reading #django 24/7 and can tell you that no matter how much we write in the docs, users are reading them less and less. So for all practical purposes a single field allowing for all of that is imo the best abstraction.

Cheers,
Florian

Shai Berger

unread,
Feb 28, 2016, 3:32:32 AM2/28/16
to django-d...@googlegroups.com
On Saturday 27 February 2016 23:57:19 Florian Apolloner wrote:
> Hi Shai,
>
> On Saturday, February 27, 2016 at 4:44:18 PM UTC+1, Shai Berger wrote:
> > I think this is why we should be a little more cautious and go for 128 or
> > 100,
> > rather than "the current maximum all-backend-compatible value".
>
> Which somewhat speaks for my point, cause we do not know if the next
> backend will need less than 100 chars (But I do agree that this is not
> likely). Tough I think that 100 (or 191 for matter) is a ridiculously low
> limit.

No, it isn't, if you think of the default char-field as a one-liner; even in
code, we don't like lines longer than 120 chars... and the default should be
friendly to backends with odd limitations, like Oracle's 8KB limitation on the
sum of inline fields in a row (which means the "max acceptable for oracle
charfield" value, which takes half of that, is a horrible default).

> In that sense, allowing max_length=None is probably the safest
> solution -- even though we do not know when database backends will crash on
> this, but we do not know this for text fields either currently.
>

The SQL Standard has many deficiencies, but it's the best resource to work
against in this regard; and the only unlimited-length character field it
supports are LOBs (see [1]). While PG treats LOBs and char-fields the same, it
is the exception rather than the rule, and I suspect it is part of the reason
for why, when push comes to shove, it does not perform as well as the
proprietary databases (don't get me wrong: I prefer it to any of them -- but
it's a bit like comparing Python with C; you pay for expressivity with
performance).

> > > That all said, if it were for me I'd deprecate CharField and only
> > > support TextField and give it an optional max_length (and a way to
> > > switch between TextArea and LineEdit for UI) -- but that is probably
> > > just because on Postgres there is no fundamental difference between them.
> > > From an enduser perspective, it is all text, the only difference is a
> > > constraint in length,
> >
> > No, that's not the only difference. There's also an understanding that
> > CharField's are one-liners, that is, there is an implicit limitation
> > (usually
> > only enforced by UI) that newlines are not allowed in the text.
>
> This is a leaky abstraction at best in my opinion. The "understanding" is
> mostly coined by the documentation in the sense that it mentions <textarea>
> for TextField and <input> for CharField. So it is kind of illogical to not
> allow a max_length on TextField.

I am not opposed to a max_length on TextField. I am opposed to setting the
default string field in Django to an unlimited-length one.

> Even with model.full_clean there is no
> restriction aside from the length, in that sense a CharField behaves
> completely the same as TextField (database issues with index lengths etc
> excluded).

Databse issues with LOBs are plenty. PG has spoiled you :) Other databases do
not do functional dependencies properly, and as a result, when you include a
LOB in a grouping or uniqune query you get either database errors (Oracle) or
terrible performance (everyone else except PG, AFAIK).

>
> For example, I don't
>
> > think there are many CharField's whose contents are interpreted as
> > markup.
>
> Maybe, maybe not, I do not see how this makes a difference here though.
> (Same goes for Regex validation).
>

The point was: They are regarded as different data types, have different
operations etc.

> > In a user perspective, these are really different data types.
>
> Which can be dangerous on it's own, I'd have to check, but I am pretty sure
> that nothing prevents the user from submitting newlines in a CharField.
>

I, too would be surprised if something does. Agreed on "leaky abstraction".

> > I find your suggestion
> > analogous to a suggestion to remove IntegerField, because "it's just a
> > FloatField with a constraint on scale".
>
> I'd suggest a DecimalField cause I do not like FloatFields either, but yeah
> ;)
>
> Do not get me wrong, I somewhat agree with everything you say (in the end I
> think it is just a matter of taste) -- but I also think that explicit is
> better than implicit.

I tend to agree with original posters in this thread -- explicit here just
means "unnecessary room for error". Django is opinionated about database
design, and I think we stand to gain by expressing on opinion about the
default length of a "regular, one-line charfield".

> From a security point of view, I agree that length
> validation should be done whenever it makes sense [which is basically
> always, there is no point in allowing to post the maximum length of a
> TextField, especially not if it contains markup which you have to render
> again ;)], so a CharField with a default max_length seems like a good idea.
> On the other hand I think that it will make the distinction for new users
> even harder. Assume we change the CharField to have a default max_length,
> the first (and probably only thing) users will notice is that CharField is
> single line. I am somewhat reading #django 24/7 and can tell you that no
> matter how much we write in the docs, users are reading them less and less.
> So for all practical purposes a single field allowing for all of that is
> imo the best abstraction.
>

Now, *that* is a leaky abstraction -- good luck hiding all the behavior
differences between limited-length and unlimited-length charfields, on any
backend but PG.

Shai.

Luke Plant

unread,
Feb 28, 2016, 3:57:27 AM2/28/16
to django-d...@googlegroups.com
Replying to this and the other emails in the thread:

Django should not be settling low arbitrary limits for the sake of a database I'm not even using, that's just crazy. Limits as high as 120 are not "big enough for anyone", and will cause problems. (I can give lots of examples). Maximum field length is a critical value that must not be left to some default, because as soon as you come to interoperate with other systems, it *does* matter.


So, this needs to be explicit.

We also need to cover the case of unlimited length Charfield for the databases that support it. This can be covered with a simple sentinel value UNLIMITED_LENGTH or similar. This is a 4th option not listed below.

We could also potentially add another sentinel like DB_MAX_UNICODE_SAFE_LENGTH that works as you expect and is clearly documented, for the sake of 3rd party apps, and comes with the caveat that it produces different behaviour on different databases.

Explicit is better than implicit etc., and in this case there is simply no sensible default behaviour that will cover all our use cases.

Luke
--
Sent from my phone

Florian Apolloner

unread,
Feb 28, 2016, 5:23:51 AM2/28/16
to Django developers (Contributions to Django itself)


On Sunday, February 28, 2016 at 9:57:27 AM UTC+1, Luke Plant wrote:
We could also potentially add another sentinel like DB_MAX_UNICODE_SAFE_LENGTH that works as you expect and is clearly documented, for the sake of 3rd party apps, and comes with the caveat that it produces different behaviour on different databases.

This is simply not an option as someone (I think Aymeric) already pointed out. Technically we do not know DB_MAX_UNICODE_SAFE_LENGTH till the call of model.save (since you can supply a using argument there), this means forms will never know the max length and could not perform any validation.

Loïc Bistuer

unread,
Feb 28, 2016, 5:33:16 AM2/28/16
to django-d...@googlegroups.com
I don't think this is a problem, we could validate that the backend supports it during save then blow up if we detect it doesn't. I think we do that for truncation on MySQL. If the model specifies something that the db doesn't support it's a configuration problem, not a user validation problem.  

Sent from my iPhone
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Florian Apolloner

unread,
Feb 28, 2016, 5:50:43 AM2/28/16
to Django developers (Contributions to Django itself)
On Sunday, February 28, 2016 at 11:33:16 AM UTC+1, Loïc Bistuer wrote:
I don't think this is a problem, we could validate that the backend supports it during save then blow up if we detect it doesn't. I think we do that for truncation on MySQL. If the model specifies something that the db doesn't support it's a configuration problem, not a user validation problem.  

I do not see how blowing up on save is a good idea (especially not if we ran model validation before). We do not do that for MySQL (at least not if your project is configured properly, leaving issues with utf8mb4 out for now) since our model form validation is able to use the supplied max_length to inform the user that the data is invalid __before__ save.

Shai Berger

unread,
Feb 28, 2016, 6:36:09 AM2/28/16
to django-d...@googlegroups.com
...or it could just mean we take the minimum of all defined databases (we would
probably like to ask the routers for the db_for_write, but this requires the
model to be defined already, so we can't do that).

No, this is still a problem: During the development, you decide to add a
second database, and all of the sudden get migrations for the whole project,
because the minimum over all defined databases changed. No good. So, I'd modify
it to something like

django.db.max_unicode_safe_length(database=DEFAULT_DB_ALIAS)

However, as I explained earlier, DB_MAX_UNICODE_SAFE_LENGTH is not a suitable
default for max_length, so this is a little besides the point here.

On Sunday 28 February 2016 10:57:17 Luke Plant wrote:
>
> Django should not be settling low arbitrary limits for the sake of a
> database I'm not even using, that's just crazy. Limits as high as 120 are
> not "big enough for anyone"

Defaults are not limits, and I don't think anybody setting a default expects
that it would fit everyone. If we did, we'd consider removing the argument
completely.

> Maximum field length is a critical value that must not be left
> to some default, because as soon as you come to interoperate with other
> systems, it *does* matter.
>
> So, this needs to be explicit.

Erg, again -- nobody is suggesting to make users unable to set their own
max_length; but many Django projects actually do not need interaction with
other systems. These are probably the majority of projects, and even a bigger
part of the projects done by beginners. We can remove an obstacle by saying,
"unless you have reason to think otherwise, we suggest that 120 is a good
length for a field intended to hold one line of text". It is an obstacle both
because beginners these days are not used to think of length limits (since the
programming language has a string type), and because they are quite likely to
get it wrong (either make it too short, or go over MySql's limit of 191).

So, to summarise: I think the database-relative constants you proposed are a
good idea; but I don't think any of them fits the bill of a default, and I do
think a default will do more good than harm.

Shai.

Loïc Bistuer

unread,
Feb 28, 2016, 9:38:57 AM2/28/16
to django-d...@googlegroups.com
There is a precedent to this (although it seems to happen at startup time rather than save time), we loop through all the connections and check that max_length has a value supported by their backend: https://github.com/django/django/blob/master/django/db/backends/mysql/validation.py.

I don’t think we need to create a new sentinel value to mean “unlimited” when `max_length=None` works well semantically, if we want to keep the `max_length` value explicit we can just use `NOT_PROVIDED` as the kwarg default value. It is backwards compatible since `max_length=None` currently issues an error.

Regarding the switch to a default value of 120 or so, I don’t have a strong opinion against it, but I think it’s an orthogonal problem to supporting unlimited varchar on backends that can handle it.

Luke Plant

unread,
Feb 28, 2016, 2:07:04 PM2/28/16
to django-d...@googlegroups.com
Regarding custom sentinels for values vs custom sentinels for missing values:

The added complication in this case is migrations, and migrations that support multiple Django versions, as is common in 3rd party apps. I haven't thought through the implications, but it is vital to do so, because getting it wrong can seriously hurt the adoption of a feature, and even the adoption of a new version of Django.

It may not be an option to say "for compatibility generate migrations using the lowest supported Django versions", because sometimes old Django versions have bugs that prevent migration generation.

We also need to consider the potential need for several different sentinel values e.g. "unlimited" vs "database defined max length".

Luke

Josh Smeaton

unread,
Feb 28, 2016, 6:59:53 PM2/28/16
to Django developers (Contributions to Django itself)
This discussion started because a user wants to leave off max_length for postgres, which allows the max_length to be omitted right? Is there anything wrong with `from django.contrib.postgres.fields import CharField`? The postgres specific CharField will allow max_length to be optional without interfering with other database backends, confusing the implementation of the standard CharField, or having to make seemingly difficult or arbitrary decisions on what a default max_length could be.

FWIW, I'm starting to come around to a default max_length of 128 for the base CharField, mainly for the reasons Shai has described (mysql max byte length for unique indexes, 128 characters being enough for the majority of CharField use-cases). There definitely are some issues with *requiring* users to arbitrarily define a max_length without having knowledge of all the ways that 255 may be a problem across backends, or understanding why 255 is even the theoretical max. If we were to consider this, I'd err on the side of changing CharField to default to max_length=128 rather than providing a new type.

Cheers

Aymeric Augustin

unread,
Feb 29, 2016, 4:19:24 AM2/29/16
to django-d...@googlegroups.com
Hello,

This thread is getting long. It contains lots of valid arguments. In the interest of seeking consensus, here’s a tentative summary.


Problem

Typing `foo = models.CharField(max_length=100)` gets old quickly. Can we skip the `max_length` argument in many cases?


Constraints

Character limits and their consequences on performance vary widely across databases, which makes it impossible for Django to provide a “large enough and fast enough for everyone” default.

The lowest limit mentioned in the thread is 191 on MySQL. No one talked about third-party backends; we can’t be sure that this value  is compatible with unique indexing on all current backends.

Making the default limit backend-dependent sounds like a bad idea, especially for pluggable applications. The implementation also looks complicated compared to the expected benefits.

PostgreSQL is a bit of an outlier. As far as I understand, there’s no performance difference VARCHAR(N) and VARCHAR(). The former just validates the length of the data.

The argument for not providing a default max_length is to have developers think about their data. Unfortunately, in practice, most just write `models.CharField(max_length=255)` and move one.

(FWIW this limit is either a cargo-cult from the 90s or a micro-optimization for MySQL.)


Consensus (?)

In order to resolve this discussion, I think we must split it in two questions.


1) Should Django provide a reasonable default for CharField.max_length?

It seems to me that the resistance to this idea weakened as the discussion unfolded. Perhaps it’s a losing fight in the long run anyway…

A default value could be justified as the “reasonable length for a line of text”. Considering that the optimal line length is around 66 characters, 100 could be a good limit. 120 would work as well.

It seems more future-proof and less debatable not to tie this limit to a technical limitation of the current version of a particular database engine.

On one hand, the default value matters because it will be widely used. On the other hand it doesn’t matter because developers who care about the length of their CharFields should set them explicitly.


2) How can we make it easy for PostgreSQL users to just use VARCHAR()?

Since this is a PostgreSQL-specific feature, having a variant of CharField in django.contrib.postgres that supports and perhaps even defaults to unlimited length shouldn’t be controversial.


I hope this helps!

-- 
Aymeric.

Shai Berger

unread,
Feb 29, 2016, 3:58:33 PM2/29/16
to django-d...@googlegroups.com
Hi,

Thank you, Aymeric, for summing up the discussion this way. The division into
two separate problems is indeed required, and I fully support the idea of
setting max_length's default to 100 or 120.

There seem to be just two points worth adding to your summary:

On Monday 29 February 2016 11:19:02 Aymeric Augustin wrote:
>
> 2) How can we make it easy for PostgreSQL users to just use VARCHAR()?
>
> Since this is a PostgreSQL-specific feature, having a variant of CharField
> in django.contrib.postgres that supports and perhaps even defaults to
> unlimited length shouldn’t be controversial.
>

The first -- I believe it was raised very early on by Christophe Pettus -- is
that Django already has a field that manifests on PG as VARCHAR(), and that is
TextField. However, I don't like the idea that PG users should be using
TextField(widget=TextInput) as a replacement for CharField; I find that
counter-intuitive -- even if just because it is a "bad name". Names are
important.

The second -- in response to a comment made by Josh Smeaton -- is that having
django.db.models.CharField with default max_lenth=N (for some finite N) and
django.contrib.postgres.CharField with default max_length=None (meaning
infinity) sounds like a bad idea.

>
> I hope this helps!

I'm certain it did!

Shai.

Cristiano Coelho

unread,
Feb 29, 2016, 7:00:29 PM2/29/16
to Django developers (Contributions to Django itself)
I find that using TextField rather than CharField just to make postgres use varchar() is a terrible idea, if you are implementing an reusable app and it is used on a backend like MySQL where TextFields are created as text columns which are horribly inneficient and should be avoided at any cost you will have a really bad time.
I'm not sure about postgres but I want to believe that using varchar without limits has also some performance considerations that should be taken care of.

Markus Holtermann

unread,
Feb 29, 2016, 8:11:33 PM2/29/16
to Django developers (Contributions to Django itself)
From what I understand you will have a hard time doing that at all:

On PG could go with a 'max_length=None' in a 3rd party app. But that wouldn't be supported on any other database backend. Which means you're limiting your app to PG. On the other hand you could make your app database independent by using a TextField which has pretty much the same performance on PG as a VARCHAR() and is available on other backends. But may perform bad on other backends. But there's no way to have a text of unlimited length on MySQL other than a TEXT column.

TL;DR: you can't achieve both ways anyway. Neither now nor if we change the behavior of Django's max_length attribute on CharFields.

/Markus

Marc Tamlyn

unread,
Mar 4, 2016, 3:05:53 PM3/4/16
to django-d...@googlegroups.com

Voting:

1) should there be a default?

I'm lazy, so I would be happy to have one. Where validation matters you can change it, where it doesn't you don't have to. I'd draw an analogy to (Positive)(Small)IntegerField - I often use the normal one when I mean some variant of validation, or some other max/min value.

2) how to make it easy to have no limit on PG?

If it's truly pg specific, then a contrib.pg field seems fine, though probably call it UnlimitedCharField for lack of ambiguity.

M

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Loïc Bistuer

unread,
Mar 4, 2016, 7:24:31 PM3/4/16
to django-d...@googlegroups.com
I’m not too keen on a contrib.pg field. CharField is the base class of many fields, a __init__ kwarg approach such as max_length=None allows us to reach those as well.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMwjO1Gj3E4_83qdbQmQwXe_WdTXUf7yuZ44-aQ9jgvJN9-5sg%40mail.gmail.com.

Shai Berger

unread,
Mar 5, 2016, 8:13:14 AM3/5/16
to django-d...@googlegroups.com
On Saturday 05 March 2016 02:24:17 Loïc Bistuer wrote:
> I’m not too keen on a contrib.pg field. CharField is the base class of many
> fields, a __init__ kwarg approach such as max_length=None allows us to
> reach those as well.
>

That's a good point; Can we enable max_length=None in a mixin?

t...@carrick.eu

unread,
Aug 12, 2020, 8:19:56 AM8/12/20
to Django developers (Contributions to Django itself)
I'd like to revive this discussion and try to come to a consensus as it's something I find myself wishing for (for Postgres in particular).

My suggestion, after reading through everything:

Give CharField a default max_length that is consistent across all vendors. It doesn't really matter what the number is other than that it should be large enough to be useful but small enough to work everywhere. I think 100 or 255 are both fine options.

If you set max_length=None explicitly, on Postgres this will use an unlimited varchar, on everything else will raise an exception on migrate.

Any thoughts?

Cheers,
Tom

Bryce Groff

unread,
Aug 16, 2020, 12:14:57 PM8/16/20
to django-d...@googlegroups.com
My 2 cents as a non core developer.  


On Wednesday, August 12, 2020, t...@carrick.eu <t...@carrick.eu> wrote:

Give CharField a default max_length that is consistent across all vendors. It doesn't really matter what the number is other than that it should be large enough to be useful but small enough to work everywhere. I think 100 or 255 are both fine options.

I think this could be a good idea, but I think it breaks the ethos of explicit is better than implicit. I can see where a user/developer would get errors (validation or db constraints) that would be very confusing as you would not know where the 100 or 255 value comes from. 
 
If you set max_length=None explicitly, on Postgres this will use an unlimited varchar, on everything else will raise an exception on migrate.

This would be very nice to have personally. I think it would need a security warning in the docs saying that this could be used as a ddos vector though. And as a potential security issue where is djangos responsibility to provide a potential footgun?


Bryce

Tom Forbes

unread,
Aug 16, 2020, 12:47:32 PM8/16/20
to django-d...@googlegroups.com
I’m not a fan of implicit max_lengths. Is having to add a keyword argument to a model field really that much of a burden? And we also would likely never be able to change the default without headaches.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Carlton Gibson

unread,
Aug 16, 2020, 2:28:47 PM8/16/20
to django-d...@googlegroups.com
Reading the history, I see an awful lot of -1s to the idea of a default. 

I see “use a TextField” and “use a subclass” a few times, which were my immediate thoughts just on the recent emails. 

Tom Carrick

unread,
Aug 17, 2020, 4:31:26 AM8/17/20
to django-d...@googlegroups.com
I'm not a fan of any solution either, really, even the one I suggested.

I think adding a new kwarg, "unlimited" seems okay to me, though it feels a little redundant.

I don't like the idea of using TextField, I find them semantically different. An unlimited varchar says to me "one possibly very long thing", whereas TextField feels more like it's free text, or a document, especially as the form fields are different. Subclassing CharField is an option, but the methods I've seen to do this makes it annoying. I need this so often that I do it all the time, but the code is so short that I don't want to bring in a new package to do it. Also, the popular ways to do this are not great. One way is to just set the max_length extremely high, which is not what I want ending up in the database, the other is something like this, which works well, but will stop working well once column collations are in as that PR adds more stuff to CharField.__init__().

I think it's time we had something in Django, whatever that ends up being.

Carlton Gibson

unread,
Aug 17, 2020, 5:02:22 AM8/17/20
to django-d...@googlegroups.com
Would the subclass in contrib.postgres suggestion be acceptable? 

Tom Carrick

unread,
Aug 17, 2020, 5:26:39 AM8/17/20
to django-d...@googlegroups.com
It would work for my use-cases. It was mentioned that it's maybe not the best as a lot of fields subclass CharField, but I don't see a compelling argument for an unlimited email or slug field. The one problem I see is CICharField, you might want to make that unlimited - but the CI fields are more or less redundant on pg >= 12 once we have column collations in. So I think it's okay for now. Only problem I can see is if we add more CharField subclasses in the future...

Adrian Torres

unread,
Oct 5, 2022, 6:47:27 AM10/5/22
to Django developers (Contributions to Django itself)
Sorry for reviving the thread but just ran into this issue again and wanted to ask, has there anything been done to address this?

For me having max_length=None mean "unlimited varchar" for Postgres and being an error for any other database (or those that don't support unlimited varchar) is the best solution, but I'm also ok with Carlton's idea of having a subclass in contrib.postgres is an acceptable compromise.

Cheers,
Adrian

Carlton Gibson

unread,
Oct 5, 2022, 8:24:35 AM10/5/22
to django-d...@googlegroups.com
Hi Adrian. 
 
Nothing has been done, no. 

There seem to be a few competing options: 

1. Allow max_length=None on Postgres (using a supports_unlimited_charfields, name!?, DB backend feature flag maybe 🤔)
2. Add an unlimited flag to CharField. 
3. Have a subclass in contrib.postgres, similar to JKM's version Tom linked before

My guess would be 1 or 3. (I'd rather 3 than 2.) So proof-of-concepts implementing those would be the way forward I think. 

Kind Regards,

Carlton


Adrian Torres

unread,
Nov 16, 2022, 2:18:21 PM11/16/22
to Django developers (Contributions to Django itself)
I finally had some time to do some work on this and have submitted a patch at https://github.com/django/django/pull/16302 in case anyone is interested.
Reply all
Reply to author
Forward
0 new messages