Supporting Oracle's renamed cx_Oracle driver in Django

[Christopher Jones]

A new, major Python cx_Oracle driver release is available and it comes with a brand new name: python-oracledb.  At run time, the module name of the package is now 'oracledb'.

With the aid of a shim to map the name space back to cx_Oracle, we  (Oracle) have run Django's tests successfully:

    import sys
    import oracledb
    oracledb.version = "8.3.0"
    sys.modules["cx_Oracle"] = oracledb
    import cx_Oracle

But the new name space could do with some love to add Django support so the shim isn't needed.  

The ideas below are from the developers who worked on the python-oracledb project and tested Django.

For background, python-oracledb is now a Thin driver by default - it connects directly to Oracle Database without always needing Oracle Client libraries.  A “Thick” mode can be optionally enabled by an application call to init_oracle_client(). This mode has similar functionality to cx_Oracle and supports Oracle Database features that extend the Python DB API.  To use this mode, the Oracle Client libraries such as from Oracle Instant Client must be installed separately - same as with cx_Oracle.

There are three main topics for python-oracledb support in Django.

1. Supporting the new name space

There are a couple of options:

1a. What about completely replacing cx_Oracle with python-oracledb (oracledb)? Python-oracledb is just a major release of cx_Oracle with a new name.  We have run Django tests.  The new driver is 'thin' by default so it's easier for almost everyone to use.

1b. Alternatively a configuration option needs to allow either cx_Oracle or python oracledb to be used.  Perhaps this could be done by creating separate sub-modules, one for each name space, inside django.db.backends.oracle? If so, then the ‘ENGINE’ values could be:

       django.db.backends.oracle.oracledb

or

       django.db.backends.oracle.cx_Oracle

2. Letting users choose whether to use python-oracledb Thin or Thick modes

The default in python-oracledb is Thin mode, which will suit almost all users.  But other people may want features available only in Thick mode.

Thick mode can only be enabled with an explicit call to init_oracle_client() before any connection is created.  Some users will need to pass a lib_dir argument to it.

Some options:

2a. It could be documented that users need to explicitly call init_oracle_client() in their own code.  This may be easiest.

2b. Or how about adding a new parameter, say ‘DRIVER_MODE’ or simply ‘MODE’, in settings.dict to allow users to use the ‘Thick’ mode of python-oracledb?  Django would need to make a call to init_oracle_client() internally.  This would need to take an optional parameter to be passed as the lib_dir argument to init_oracle_client()

3. Supporting some new connection parameters and using service names instead of SIDs

One new feature of the python-oracledb driver are additional connect() keyword arguments like hostname, port, service_name, tcp_connect_timeout.  (The makedsn() function is now deprecated in python-oracle because of this change.)  How can new parameters best be passed to python-oracledb?

It may also be time to modernize some existing connection code in Django for Oracle. For example, to more obviously connect using database 'service names' instead of SIDs (which were obsoleted decades ago).  We recommend using service names instead of SIDs so that various database properties like Oracle's Application Continuity feature can be enabled.

It would be nice if apps could do something like:

   DATABASES = {
       'default': {
           'ENGINE': 'django.db.backends.oracle',
           'USER': 'scott',
           'PASSWORD': 'XXXX',
           'OPTIONS': {
               'host': 'example.com',
               'port': 1521,
               'service_name': 'orclpdb',
               'tcp_connect_timeout': 10,
          },
       }
   }

We have tossed around various ideas about all these but know it's best to get Django community involvement early about the design.

Chris

[Florian Apolloner]

Hi Chris,

a thin client seems like great news to me :)

On Friday, May 27, 2022 at 3:05:51 PM UTC+2 christopher....@gmail.com wrote:

1a. What about completely replacing cx_Oracle with python-oracledb (oracledb)? Python-oracledb is just a major release of cx_Oracle with a new name.  We have run Django tests.  The new driver is 'thin' by default so it's easier for almost everyone to use.

Assuming the differences between the two drivers are not to big I'd would be great if the existing backend would support both names for now. Then we can drop the cx_oracle imports relatively easily after a release or two.

 

1b. Alternatively a configuration option needs to allow either cx_Oracle or python oracledb to be used.  Perhaps this could be done by creating separate sub-modules, one for each name space, inside django.db.backends.oracle? If so, then the ‘ENGINE’ values could be ...

I am against this, if python-oracledb is the new cx_oracle I guess cx_oracle won't really be developed anymore (correct me if I am wrong) and as such I do not see much point in supporting both. Oracle is a beast on it's own, one oracle driver to handle is more than enough :)

2. Letting users choose whether to use python-oracledb Thin or Thick modes

2a. It could be documented that users need to explicitly call init_oracle_client() in their own code.  This may be easiest.

Would be fine for me if the testsuite passes with thin mode.

 

2b. Or how about adding a new parameter, say ‘DRIVER_MODE’ or simply ‘MODE’, in settings.dict to allow users to use the ‘Thick’ mode of python-oracledb?  Django would need to make a call to init_oracle_client() internally.  This would need to take an optional parameter to be passed as the lib_dir argument to init_oracle_client()

Should be fine I think, lets see what others think about that. But the config approach would certainly be the preferred way of doing things over the documentation approach (in the longrun at least). Then again I am not really sure which features thick mode offers and whether Django developers will actually need that often… You probably know that better than I do.

3. Supporting some new connection parameters and using service names instead of SIDs

One new feature of the python-oracledb driver are additional connect() keyword arguments like hostname, port, service_name, tcp_connect_timeout.  (The makedsn() function is now deprecated in python-oracle because of this change.)  How can new parameters best be passed to python-oracledb?

I'd say use get_connection_params & get_new_connection https://github.com/django/django/blob/8c0886b068ba4e224dd78104b93c9638b860b398/django/db/backends/oracle/base.py#L245-L259 -- or do I miss something here.

It would be nice if apps could do something like:

   DATABASES = {
       'default': {
           'ENGINE': 'django.db.backends.oracle',
           'USER': 'scott',
           'PASSWORD': 'XXXX',
           'OPTIONS': {
               'host': 'example.com',
               'port': 1521,
               'service_name': 'orclpdb',
               'tcp_connect_timeout': 10,
          },
       }
   }

I think obvious keys like host & port should be kept in the toplevel like we already do https://docs.djangoproject.com/en/4.0/ref/databases/#connecting-to-the-database-1 and the NAME could become the service_name (though I do see how this could clash with the current usage)? Then again we are not 100% consistent all the time here, so whatever feels more natural I guess?

I hope this helps somewhat. Let us know if there is more you'd like to know.

Cheers,

Florian

[Mariusz Felisiak]

I agree with Florian, I'd prefer adding support for python-oracledb in Django 4.1 and immediately deprecate using cx_Oracle (will be removed in Django 5.0).

3. Supporting some new connection parameters and using service names instead of SIDs

One new feature of the python-oracledb driver are additional connect() keyword arguments like hostname, port, service_name, tcp_connect_timeout.  (The makedsn() function is now deprecated in python-oracle because of this change.)  How can new parameters best be passed to python-oracledb?

As far as I'm aware this is already supported with an easy connect string or full DSN in NAME, see docs.

Best,
Mariusz

[Christopher Jones]

> I agree with Florian, I'd prefer adding support for python-oracledb in Django 4.1 and immediately deprecate using cx_Oracle (will be removed in Django 5.0).

That sounds fair.  The cx_Oracle namespace won't have any substantive changes; maybe some new wheels for Python 3.11, and any critical bug fixes.  All progress will be under the new name.

The question is how to add support cleanly so that both names are supported in 4.1?  Is there a preference?  Particularly how can it be be done to reuse code without (temporary) duplication?

Regarding connection options, yes some things are supported with the Easy Connect Plus syntax.  There are some connection properties that aren't supported, e.g  application contexts (this was also true of cx_Oracle). And maybe more in future depending what users ask for. A generic way to set these, e.g with OPTIONS is a good direction.  

Chris

[Florian Apolloner]

Hi Chris,

On Tuesday, June 7, 2022 at 11:30:46 AM UTC+2 christopher....@gmail.com wrote:

The question is how to add support cleanly so that both names are supported in 4.1?  Is there a preference?  Particularly how can it be be done to reuse code without (temporary) duplication?

Good question, no idea :) I am having the same problem with psycopg3 and this is what I did https://github.com/django/django/pull/15687/files#diff-01f6880f77beca32ee83e011072ba73dc7eed7f9f3efdebd935af693a4fac7b3 -- basically I added a compatibility module which does imports from the respective locations, with the idea of being able to simply run sed over this in the future. I know this isn't much of an answer and the differences between psycopg2 & 3 are rather minimal but maybe a similar approach is viable for oracle as well?

 

Regarding connection options, yes some things are supported with the Easy Connect Plus syntax.  There are some connection properties that aren't supported, e.g  application contexts (this was also true of cx_Oracle). And maybe more in future depending what users ask for. A generic way to set these, e.g with OPTIONS is a good direction.  

Yes, I'd use the existing backends as guidelines, common stuff like HOST etc fits into the toplevel imo.

Cheers,

Florian

[Christopher Jones]

Thanks Florian - we'll try out some ideas.

Chris