python manage.py runserver can not use newly compiled sqlite3 lib on CentOS 7

[panfei]

System environment:

Cent OS 7

Sqlite3 3.34.0 (Compile from source)

Python 3.9.1 (Compile from source)

Django 3.1.5 (Pip install)

1. Compile sqlite3:

./configure --prefix=/home/felix/.local/sqlite/sqlite-3.34.0

make && make install

2. Add sqlite3 lib to lib search path:

export LD_LIBRARY_PATH=/home/felix/.local/sqlite/default/lib

export LD_RUN_PATH=/home/felix/.local/sqlite/default/lib

3. Compile Python 3.9.1

C_INCLUDE_PATH=/home/felix/.local/sqlite/sqlite-3.34.0/include/ CPLUS_INCLUDE_PATH=/home/felix/.local/sqlite/sqlite-3.34.0/include/ LD_RUN_PATH=/home/felix/.local/sqlite/default/lib ./configure --prefix=/home/felix/.local/python/python-3.9.1 --enable-optimizations

make && make install

4. Create a venv and install django and start a demo project

cd /tmp

/home/felix/.local/python/python-3.9.1/bin/python3 -m venv venv

source venv/bin/activate

pip install django

djagno-admin startproject demo

cd demo

4 Check sqlite3 lib version (seems to be OK, sqlite_version is 3.34.0)

(venv) [felix@localhost blueprint]$ python manage.py shell
Python 3.9.0 (default, Jan 20 2021, 12:53:25)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.34.0'
>>> sqlite3.version
'2.6.0'
>>>

5. Run the demo project (cannot find the new sqlite version ? strange.):

(venv) [felix@localhost blueprint]$ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 215, in get_new_connection
    create_deterministic_function('django_date_extract', 2, _sqlite_datetime_extract)
sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/felix/.local/python/python-3.9.0/lib/python3.9/threading.py", line 950, in _bootstrap_inner
    self.run()
  File "/home/felix/.local/python/python-3.9.0/lib/python3.9/threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 121, in inner_run
    self.check_migrations()
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/core/management/base.py", line 459, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/migrations/loader.py", line 216, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
    if self.has_table():
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 55, in has_table
    with self.connection.cursor() as cursor:
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/felix/PycharmProjects/blueprint/venv/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 215, in get_new_connection
    create_deterministic_function('django_date_extract', 2, _sqlite_datetime_extract)
django.db.utils.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher

Anyone can give help on this issue. Thanks .

--

Code everyday.

[panfei]

I found that, it may not be a Django-related issue, It's more likely a Python level issue, but do anyone know how to avoid this problem?:

(venv) [felix@localhost blueprint]$ python

Python 3.9.1 (default, Jan 20 2021, 14:32:50)

[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sqlite3
>>> conn = sqlite3.connect(':memory:')
>>> conn.create_function('f', 2, lambda *args: None, deterministic=True)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher

>>> sqlite3.sqlite_version
'3.34.0'
>>> sqlite3.version
'2.6.0'
>>>

panfei <cnw...@gmail.com> 于2021年1月20日周三 下午8:03写道：

--

不学习，不知道

[Amitesh Sahay]

Try older version of Python interpreter and see if that works. May be  consider using 3.6

Regards,

Amitesh 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/django-users/CA%2BJstLDBJ1xR1PsDNJDQKMQi%2B_aZZxn41FW%2BevgCRGzsQjQiuA%40mail.gmail.com

.

[panfei]

I have tried, it works in Python 3.6 environment. but I still want to find a way (maybe a workaround) to solve this problem.

'Amitesh Sahay' via Django users <django...@googlegroups.com> 于2021年1月20日周三 下午9:48写道：

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1538545520.285499.1611150495253%40mail.yahoo.com.

--

不学习，不知道

[panfei]

The exception is raised from inside the C implementation in Python when checking sqlite3 version check.

In Django:

@async_unsafe
def get_new_connection(self, conn_params):
    conn = Database.connect(**conn_params)
    if PY38:
        create_deterministic_function = functools.partial(
            conn.create_function,
            deterministic=True,
        )
    else:
        create_deterministic_function = conn.create_function

if PY38 (means python version greater or equal to 3.8) is True, the new created function will call conn.create_function with deterministic=True, so the exception happens.

panfei <cnw...@gmail.com> 于2021年1月20日周三 下午9:54写道：

--

不学习，不知道

[Amitesh Sahay]

So, this seems to be a version compatibility issue, a a work around you can use some other database(mysql, oracle, etc). This cud save ur day and time as well.

Regards,

Amitesh 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2BJstLB6ihG8C%2Br-4fjkt2k%3DhLCK8R-rm4uJ2TViBY70G8G5vw%40mail.gmail.com.

[Amitesh Sahay]

Please read the release notes of the Django version you are using. It will be a life saver as well.

Regards,

Amitesh 

[panfei]

Thanks for all the suggestions.

'Amitesh Sahay' via Django users <django...@googlegroups.com> 于2021年1月20日周三 下午10:06写道：

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1524829928.2923252.1611151558687%40mail.yahoo.com.

--

不学习，不知道

[panfei]

It finally proved a compilation issue, this is the right way to compile Python 3.9:

1033 C_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ CPLUS_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ LD_RUN_PATH=/home/felix/.local/sqlite/default/lib ./configure --prefix=/home/felix/.local/python/python-3.9.1 --enable-optimizations
1034 C_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ CPLUS_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ LD_RUN_PATH=/home/felix/.local/sqlite/default/lib make
1035 make clean
1036 C_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ CPLUS_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ LD_RUN_PATH=/home/felix/.local/sqlite/default/lib ./configure --prefix=/home/felix/.local/python/python-3.9.1 --enable-optimizations
1037 C_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ CPLUS_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ LD_RUN_PATH=/home/felix/.local/sqlite/default/lib make
1038 C_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ CPLUS_INCLUDE_PATH=/home/felix/.local/sqlite/default/include/ LD_RUN_PATH=/home/felix/.local/sqlite/default/lib make install
1039 /home/felix/.local/python/python-3.9.1/bin/python3
1040 history
[felix@localhost Python-3.9.1]$ /home/felix/.local/python/python-3.9.1/bin/python3
Python 3.9.1 (default, Jan 21 2021, 10:58:50)

[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> conn = sqlite3.connect(':memory:')
>>> conn.create_function('f', 2, lambda *args: None, deterministic=True)
>>>

Every step should specify the same environment variables.

'Amitesh Sahay' via Django users <django...@googlegroups.com> 于2021年1月20日周三 下午10:06写道：

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1524829928.2923252.1611151558687%40mail.yahoo.com.

--

不学习，不知道

[Joseph Borge]

I tried to configure and compile with all 3 variables and that did work as expected.
Thanks!

# C_INCLUDE_PATH=/usr/local/include CPLUS_INCLUDE_PATH=/usr/local/include LD_RUN_PATH=/usr/local/lib ./configure --enable-optimizations

# C_INCLUDE_PATH=/usr/local/include CPLUS_INCLUDE_PATH=/usr/local/include LD_RUN_PATH=/usr/local/lib make

# C_INCLUDE_PATH=/usr/local/include CPLUS_INCLUDE_PATH=/usr/local/include LD_RUN_PATH=/usr/local/lib make install