CharField with Oracle Backend Null Behaviour

192 views
Skip to first unread message

vaf...@exscientia.co.uk

unread,
Nov 8, 2018, 7:17:30 AM11/8/18
to Django developers (Contributions to Django itself)
Dear Django Team,

CharField on an Oracle backend has some strange behaviour. Because oracle treats empty strings and nulls in a similar fashion, Django simply ignore the null parameter (which should control nullability) on the field.

I agree that an empty string and a null value are under most circumstances interchangeable, but this is being conflated with the concept of nullability.
Nullability has nothing to do with the representation of the null, just wether or not the field may be null.
i.e If a column has a not null constraint, it should not accept an empty string or 'Null' 
With Django's current behaviour, since declaring a field as not null has no effect, both 'Null' and blank strings may be inserted without issue.
I would argue that this is a broken design. How are raw queries enforced, or additional clients that have no knowledge of the Django's internal checks and conversions? 
If Django needs to do internal checks, it should use the value of 'blank' and not 'null' to be consistent with other backends. Oracle supports not null constraints, why not exploit that?

If this would be accepted, would be happy to create a PR. What are your thoughts?

Many Thanks,
Vackar

Florian Apolloner

unread,
Nov 8, 2018, 7:43:25 AM11/8/18
to Django developers (Contributions to Django itself)
Oracle treats NULL and empty varchar2 the same; so null=True/False is simply not possible on Oracle for CharField. I am not sure what you are proposing here…

Václav Řehák

unread,
Nov 8, 2018, 8:10:30 AM11/8/18
to Django developers (Contributions to Django itself)

While it is probably not possible to change how Django treats this, many newcomers find it super confusing. To make developer experience in our Oracle-based project better, we started using the following workaround for fields which are required to be not null and not empty string:

class NonEmptyCharField(models.CharField):
    """
    CharField preventing empty string and null

    It workarounds two problems:

    - CharField has a default value '' which allows silent save of model without an error even
      if we forget to set mandatory fields. This is solved by setting the default to None causing
     db error on not null constraint.

    - The above is not sufficient for Oracle as the db backend has hardcoded null=True regardless
      of what we set. This is changed by empty_strings_allowed=False.
    """
    empty_strings_allowed = False

    def __init__(self, *args, **kwargs):
        kwargs.setdefault('default', None)
        super().__init__(*args, **kwargs)

Dne čtvrtek 8. listopadu 2018 13:43:25 UTC+1 Florian Apolloner napsal(a):

vaf...@exscientia.co.uk

unread,
Nov 8, 2018, 9:52:01 AM11/8/18
to Django developers (Contributions to Django itself)
Hi Florian,

Thanks for getting back to me. Allow me elaborate
My main question is: what is the rationale for enforcing null value checks in middleware, and not delegating to DB?

My proposal would be:
- If null=False is specified, then add an explicit not null constraint at the db level
- When converting between python and sql, convert empty strings to nulls

This would prevent insertion of blank strings and nulls into required columns, regardless of the client accessing the DB - something that can currently only be mimicked at the middleware level, and requires all clients to use Django. 

Hope this makes things clearer.

Thanks,
Vackar

vaf...@exscientia.co.uk

unread,
Nov 8, 2018, 10:05:14 AM11/8/18
to Django developers (Contributions to Django itself)
Vaclav, this is an interesting approach. I know it's a very simple field, but have you though of creating an open source package for this, looks really useful.
Would probably call it RequiredCharField though, the double negative in NonEmptyCharField can make it a slightly harder to read.

Florian Apolloner

unread,
Nov 8, 2018, 11:10:34 AM11/8/18
to Django developers (Contributions to Django itself)
On Thursday, November 8, 2018 at 3:52:01 PM UTC+1, vaf...@exscientia.co.uk wrote:
- If null=False is specified, then add an explicit not null constraint at the db level

As far as I understand Oracle makes no difference between null and an empty string. So if we were to add a not-null constraint, you could also no longer insert an empty string; which seems counterintuitive. Note that blank=True/False should be used there as validation on whether or not you want to allow empty values.

This would prevent insertion of blank strings and nulls into required columns, regardless of the client accessing the DB

Insertion of blank string and null is prevent if you set blank=False and run model validation, isn't it?

I have to say my Oracle knowledge is somewhat limited, so please be patient and explain a bit more if I missunderstand you completly (code examples where you think the behavior is wrong would also help).

Cheers,
Florian

vaf...@exscientia.co.uk

unread,
Nov 8, 2018, 11:36:53 AM11/8/18
to Django developers (Contributions to Django itself)
Hi Florian

No probs, most people don't understand Oracle, it's a bit of a black art. Some background: Partly for legacy and partly for justifiable reasons, oracle converts blank strings to null.
Back to this issue, as you say, if you implemented the above recommendations you would not be able to insert a blank string into required fields, but this would be 100% in line with how Oracle works anyway. Some DBs handle this differently, and I don't want to start a debate around which is better / correct, but my feeling is that one should go with the model as described by the DB. i.e. if using Postgres, use the Postgres model, when working with Oracle, use the oracle model etc. 

My main concern currently is that required fields are not enforced at the db level, which makes using it with other clients difficult. I would much prefer that constraints be added, and accept that empty strings cannot be inserted into required columns.

Let me know what you think,
Vackar

vaf...@exscientia.co.uk

unread,
Nov 8, 2018, 11:53:32 AM11/8/18
to Django developers (Contributions to Django itself)
Vaclav,

With your approach, fields are correctly created with / without null constraints based on the value of the 'null' parameter.
However subsequent changes are not being picked up by the migrations framework. i.e. if I change null=True to null=False, constraint is not dropped. The same is true going from False to True, the constraint is not added. Did you have to monkey patch the migrations module to get it to work, or do you have some other approach?

Many Thanks,
Vackar

On Thursday, 8 November 2018 13:10:30 UTC, Václav Řehák wrote:

Florian Apolloner

unread,
Nov 8, 2018, 12:06:05 PM11/8/18
to Django developers (Contributions to Django itself)
Hi Vackar,

Thank you, now we are getting somewhere!

On Thursday, November 8, 2018 at 5:36:53 PM UTC+1, vaf...@exscientia.co.uk wrote:
My main concern currently is that required fields are not enforced at the db level, which makes using it with other clients difficult. I would much prefer that constraints be added, and accept that empty strings cannot be inserted into required columns.

Okay, now I get you. I have no strong feelings how our oracle backend should behave here; other than your suggested change would be (highly?) backwards incompatible and that alone might prevent it from getting merged. Maybe Felix can chime in here with his oracle knowledge.

FWIW, Django has quite a few places where it does stuff in the framework itself instead of at the database level. Examples include the handling of database default values (we don't, defaults are in the application, even for simple integers and strings) as well as handling of cascades on deletes. If one cannot live with those issues, it is imo currently best to manage the database independent from Django. We are certainly open to improving in those areas though.

Cheers,
Florian


vaf...@exscientia.co.uk

unread,
Nov 8, 2018, 12:33:52 PM11/8/18
to Django developers (Contributions to Django itself)
Hi Florian,

Yes you are correct, backwards compatibility would break with this change. 
As Django already mimics the required checks in middleware for nulls, only difference I can see going forward would be that newly inserted blank strings would be read back back as None, instead of '' if nulls are allowed. And where nulls are not allowed, neither would make it into the db.

Look forward to hearing your thoughts Felix.

Many Thanks,
Vackar

Mariusz Felisiak

unread,
Nov 8, 2018, 2:14:01 PM11/8/18
to Django developers (Contributions to Django itself)
Hi,

    Oracle treats empty strings and NULLs in the same way, that's why we had to decide on some workaround that will cover both python's (other Dbs') cases i.e. NULLs and "" (see https://docs.djangoproject.com/en/2.1/ref/databases/#null-and-empty-strings). I don't see much value in changing current behavior from "empty strings always" to "NULLs always", because it will be backward incompatible and IMO it doesn't change anything, still non-Oracle people will be surprised by this behavior.
    I agree that creating a "NOT NULL" constraint in case when "null=False" (or even "blank=False") may be a helpful addition to the current behavior but I'm not sure how doable it is if we take into account migration of existing projects.

Best,
Mariusz

Dan Davis

unread,
Nov 8, 2018, 9:57:09 PM11/8/18
to django-d...@googlegroups.com
Part of the reason to use Django is to assure some level of database portability.  I use Oracle exclusively at work, although there are some plans to move towards PostgreSQL.  I think blank=True should govern whether the field may be null or not.

So, from my perspective, we should limit ourselves with tinkering with how the backend represents such constraints at the database level.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/403d5d50-660b-426c-a911-579d82273259%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

vaf...@exscientia.co.uk

unread,
Nov 12, 2018, 3:58:33 AM11/12/18
to Django developers (Contributions to Django itself)
Hi Mariusz,

Thanks for getting back to me, didn't realise that default was empty strings, that would work too. With regards to existing projects, migration should be really straightforward, just need to add a constraint to db on the next migration. It would generate migrations where no model changes have been made, but I would be comfortable with that, just need to document it properly. 

Is there a mecanhism by which we can propose this change in a more formal manner?

Many Thanks,
Vackar

Dan Davis

unread,
Feb 13, 2019, 10:13:48 PM2/13/19
to Django developers (Contributions to Django itself)
This very issue of a not-null constraint causes a problem with django-silk.  I'm not sure this matters in practice, because the usage of django silk is usually for development; I just added an sqlite3 database alias and a router for django-silk.   However, it validates the problem of the current approach.
Reply all
Reply to author
Forward
0 new messages