[Django] #20968: Error creating Indexes on Syncdb
Django
------------------------------+--------------------------------------------
Reporter: | Owner: nobody
burton449geo@… | Status: new
Type: Bug | Version: 1.5
Component: GIS | Keywords: geodjango, spatialite, indexes
Severity: Normal | Has patch: 0
Triage Stage: Unreviewed | UI/UX: 0
Easy pickings: 0 |
------------------------------+--------------------------------------------
The database is Spatialite 4.1.0
On Syncdb these errors messages are shown:
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
AddGeometryColumn() error: "no such table: geometry_columns"
CreateSpatialIndex() error: "no such table: geometry_columns"
--
Ticket URL: <https://code.djangoproject.com/ticket/20968>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: burton449geo@… | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.5
Severity: Normal | Resolution:
Keywords: geodjango, | Triage Stage:
spatialite, indexes | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by leplatrem):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
Initializing the database with spatialite binary is the answer :
spatialite database.db "SELECT InitSpatialMetaData();"
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:1>
Django
Reporter: burton449geo@… | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.5
Severity: Normal | Resolution:
Keywords: geodjango, | Triage Stage:
spatialite, indexes | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by claudep):
This should be done automatically:
https://github.com/django/django/blob/master/django/contrib/gis/db/backends/spatialite/creation.py#L102-L107
Maybe something changed in Spatialite 4.1? I cannot test right now.
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:2>
Django
Reporter: burton449geo@… | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.5
Severity: Normal | Resolution:
Keywords: geodjango, | Triage Stage:
spatialite, indexes | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by aaugustin):
Since no one managed to reproduce the error, should I close this ticket as
needsinfo?
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:3>
Django
Reporter: burton449geo@… | Owner: nobody
Component: GIS | Version: 1.5
Severity: Normal | Resolution: needsinfo
Keywords: geodjango, | Triage Stage:
spatialite, indexes | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => needsinfo
Comment:
Tried to reproduce this to not avail (tested with the !GeoDjango tutorial
project).
Environment:
- Ubuntu 12.04 LTS
- System-wide Linux-distribution-provided packages of GDAL, PROJ, geoip,
GEOS, SQLite3
- Manually built inside a Python 2.7 virtualenv:
- spatialite 4.1.0
- spatialite-tools 4.1.0
- pysqlite2 2.6.3 with a modified `setup.cfg` as per
https://docs.djangoproject.com/en/1.6/ref/contrib/gis/install/spatialite/#pysqlite2
- Django master as of now
Creating spatial metadata tables (per
https://docs.djangoproject.com/en/dev/ref/contrib/gis/install/spatialite/#creating-a
-spatial-database-for-spatialite) -- Not strictly needed but useful to get
the versions of the C-level components printed to the console:
{{{
(spatialite41)ramiro@mang:~/venv/spatialite41/src/geodjango$ spatialite
db.sqlite3 "SELECT InitSpatialMetaData();"
SpatiaLite version ..: 4.1.0 Supported Extensions:
- 'VirtualShape' [direct Shapefile access]
- 'VirtualDbf' [direct DBF access]
- 'VirtualText' [direct CSV/TXT access]
- 'VirtualNetwork' [Dijkstra shortest path]
- 'RTree' [Spatial Index - R*Tree]
- 'MbrCache' [Spatial Index - MBR cache]
- 'VirtualSpatialIndex' [R*Tree metahandler]
- 'VirtualFDO' [FDO-OGR interoperability]
- 'SpatiaLite' [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.7.1, 23 September 2009
GEOS version ........: 3.2.2-CAPI-1.6.2
the SPATIAL_REF_SYS table already contains some row(s)
InitSpatiaMetaData() error:"table spatial_ref_sys already exists"
0
}}}
DB schema manage.py commands:
{{{
(spatialite41)ramiro@mang:~/venv/spatialite41/src/geodjango$
PYTHONPATH=~/django/upstream python manage.py sqlall world
BEGIN;
CREATE TABLE "world_worldborder" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"name" varchar(50) NOT NULL,
"area" integer NOT NULL,
"pop2005" integer NOT NULL,
"fips" varchar(2) NOT NULL,
"iso2" varchar(2) NOT NULL,
"iso3" varchar(3) NOT NULL,
"un" integer NOT NULL,
"region" integer NOT NULL,
"subregion" integer NOT NULL,
"lon" real NOT NULL,
"lat" real NOT NULL
)
;
SELECT AddGeometryColumn('world_worldborder', 'mpoly', 4326,
'MULTIPOLYGON', 2, 1);
SELECT CreateSpatialIndex('world_worldborder', 'mpoly');
COMMIT;
}}}
{{{
(spatialite41)ramiro@mang:~/venv/spatialite41/src/geodjango$
PYTHONPATH=~/django/upstream python manage.py migrate
Operations to perform:
Synchronize unmigrated apps: gis, sessions, admin, messages, auth,
staticfiles, contenttypes, world
Apply all migrations: (none)
Synchronizing apps without migrations:
Creating tables...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table world_worldborder
Installing custom SQL...
Installing indexes...
Installed 0 object(s) from 0 fixture(s)
Running migrations:
No migrations needed.
You just installed Django's auth system, which means you don't have any
superusers defined.
Would you like to create one now? (yes/no): no
}}}
Also ran the !GeoDjango tests without problems:
{{{
(spatialite41)ramiro@mang:~/django/upstream/tests$ PYTHONPATH=.. python
runtests.py --settings=test_gis django.contrib.gis
Testing against Django installed in '/home/ramiro/django/upstream/django'
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.s...s....s..s............s.......sssssssssss..........ssssss...s............s....s.......sss...s.s.s.s.......ssssss..........................................................................................................................................................................................
----------------------------------------------------------------------
Ran 302 tests in 5.177s
OK (skipped=38)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...
}}}
If you have a system-wide spatialite installation of a different version
(e.g. in the case of Ubuntu 12.04 it could be the 3.0.0~beta20110817-3
.deb package) then make sure you direct !GeoDjango to use the 4.1.x one
using the `SPATIALITE_LIBRARY_PATH` setting in setting.py:
{{{
SPATIALITE_LIBRARY_PATH='/path/to/your/spatialite/4.1.x/libspatialite.so'
}}}
Closing 'needsinfo' for now. Please reopen with a similar description of
your environment if you can reproduce and wish to push this further.
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:4>
Django
Reporter: burton449geo@… | Owner: nobody
Type: Bug | Status: closed
Component: GIS | Version: 1.5
Keywords: geodjango, | Triage Stage:
spatialite, indexes | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by ramiro):
* resolution: needsinfo => invalid
Comment:
Actually I can reproduce the issue if I don't create the spatial tables
with `spatialite db.sqlite3 "SELECT InitSpatialMetaData();"` then the
index creation fails.
So comment:1 is right. I'm wrong in the last comment because I assumed
these tables were created automatically if not present. That's only true
then creating the test DB to run the !GeoDjango tests.
So what one needs to do is actually follow the documentation
(https://docs.djangoproject.com/en/dev/ref/contrib/gis/install/spatialite/#creating-a
-spatial-database-for-spatialite) because at no point it describes that
step as optional.
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:5>
Django
Reporter: burton449geo@… | Owner: nobody
Type: Bug | Status: closed
Component: GIS | Version: 1.5
Severity: Normal | Resolution: invalid
Keywords: geodjango, | Triage Stage:
spatialite, indexes | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by txomon):
It does specify that step as optional in 1.7 docs:
https://docs.djangoproject.com/en/1.7/ref/contrib/gis/install/spatialite/#creating-a
-spatial-database-for-spatialite
Maybe an update would be recommended?
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:6>
Django
Reporter: burton449geo@… | Owner: nobody
Type: Bug | Status: closed
Component: GIS | Version: 1.5
Severity: Normal | Resolution: invalid
Keywords: geodjango, | Triage Stage:
spatialite, indexes | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by kenial):
Using `spatialite db.sqlite3 "SELECT InitSpatialMetaData();"` is the
answer, though, however, can we prevent Django users from meeting this
error by adding workaround codes, which would be in syncdb command or
database file creation code of django.contrib.gis.db.backends.spatialite?
p.s: txomon, I'm one of them recently updated it and feel sorry. As you
know, using spatialite is something... feels like rolling under the hood
:(
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:7>
Django
Reporter: burton449geo@… | Owner: nobody
Cleanup/optimization | Version: master
Component: GIS | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by claudep):
* status: closed => new
* resolution: invalid =>
* version: 1.5 => master
* has_patch: 0 => 1
* type: Bug => Cleanup/optimization
* stage: Unreviewed => Accepted
Comment:
What about this patch?
https://github.com/django/django/pull/3697
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:8>
Django
Reporter: burton449geo@… | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: GIS | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by kenial):
Replying to [comment:8 claudep]:
> What about this patch?
> https://github.com/django/django/pull/3697
if InitSpatialMetaData query is executed in init_connection_state()
method, then it will check it **every time when opening a connection**. As
you know, there is no connection pooling on sqlite3 (and SpatiaLite)
backend by default. It means that every query execution will lead to
execute `PRAGMA table_info(geometry_columns);`, I think it's a sort of
redundancy. My patch updating django/contrib/gis/apps.py intended to
execute InitSpatialMetaData query when doing **migrate**:
https://github.com/django/django/pull/3695
What do you think about?
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:9>
Django
Reporter: burton449geo@… | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: GIS | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* cc: keniallee@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:10>
Django
Reporter: burton449geo@… | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: GIS | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by claudep):
I recognize my patch has the drawback of adding a query for each database
connection creation.
What I don't like in your solution, it's the backend-specific "pollution"
in the main gis code. If we miss a backend-specific hook during app
initialization, then let's create that hook. But importing anything from
`django.contrib.gis.db.backends.<backend>` inside `apps.py` is not an
option.
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:11>
Django
Reporter: burton449geo@… | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: GIS | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by claudep):
Another option would be to use the
[https://docs.djangoproject.com/en/dev/ref/signals/#pre-migrate pre-
migrate] signal.
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:12>
Django
Reporter: burton449geo@… | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: GIS | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by kenial):
If we're going with the hook, it still does "migrate" stuff in context of
another application (in this case, gis). So hook isn't looking good. (is
there a way to completely split into migrate and gis context in app
initialization?)
pre-migrate signal seems to be nice, but I find out ... it's `model`
based. We're doing this on database-level. Maybe it's safe to be executed
repeatedly, because when migrating only.
My original approach was to add a method for this purpose (name was
`before_migration`) to BaseDatabaseWrapper with empty implementation, and
call it when running `migrate` command. For sure,
`django.contrib.gis.db.backends.spatialite.base.DatabaseWrapper`
implements this `before_migration` method to init metadata table.
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:13>
Django
Reporter: burton449geo@… | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: GIS | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by kenial):
Here's my original approach: https://github.com/django/django/pull/3698
How about this?
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:14>
Django
Reporter: burton449geo@… | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
Component: GIS | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by claudep):
I've rewritten the patch with the migrate hook idea. I also added a patch
which moves a PostGIS operation to add some weight to this solution.
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:15>
Django
-------------------------------------+-------------------------------------
Reporter: burton449geo@… | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: master
(models, ORM) | Triage Stage: Ready for
Severity: Normal | checkin
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* component: GIS => Database layer (models, ORM)
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:16>
Django
-------------------------------------+-------------------------------------
Reporter: burton449geo@… | Owner: nobody
Cleanup/optimization | Version: master
Component: Database layer | Resolution: fixed
(models, ORM) | Triage Stage: Ready for
Severity: Normal | checkin
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"8f97413faed5431713c034897cda486507bf0cc3"]:
{{{
#!CommitTicketReference repository=""
revision="8f97413faed5431713c034897cda486507bf0cc3"
Fixed #20968 -- Checked Spatialite metadata before migrations
Thanks Kenial S. Lee for the initial patch and Tim Graham for
the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:17>
Django
-------------------------------------+-------------------------------------
Reporter: burton449geo@… | Owner: nobody
Type: | Status: closed
Cleanup/optimization | Version: master
Component: Database layer | Resolution: fixed
(models, ORM) | Triage Stage: Ready for
Severity: Normal | checkin
Keywords: geodjango, | Needs documentation: 0
spatialite, indexes | Patch needs improvement: 0
Has patch: 1 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by kenial):
Great work! Thanks :)
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:18>
Django
-------------------------------------+-------------------------------------
Reporter: burton449geo@… | Owner: nobody
Type: | Status: closed
Cleanup/optimization | Version: master
Component: Database layer | Resolution: fixed
Severity: Normal | Needs documentation: 0
Keywords: geodjango, | Patch needs improvement: 0
spatialite, indexes | UI/UX: 0
Has patch: 0 |
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* has_patch: 1 => 0
* stage: Ready for checkin => Accepted
Comment:
Claude, PostGIS on the django-master-trusty build is failing since this
change. Could you take a look?
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:19>
Django
-------------------------------------+-------------------------------------
Reporter: burton449geo@… | Owner: nobody
Type: | Status: closed
Cleanup/optimization | Version: master
Component: Database layer | Resolution: fixed
(models, ORM) | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: geodjango, | Patch needs improvement: 0
spatialite, indexes | UI/UX: 0
Has patch: 0 |
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by claudep):
Tentative fix: https://github.com/django/django/pull/3712
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:20>
Django
-------------------------------------+-------------------------------------
Reporter: burton449geo@… | Owner: nobody
Type: | Status: closed
Cleanup/optimization | Version: master
Component: Database layer | Resolution: fixed
(models, ORM) | Triage Stage: Accepted
Severity: Normal | Needs documentation: 0
Keywords: geodjango, | Patch needs improvement: 0
spatialite, indexes | UI/UX: 0
Has patch: 0 |
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by Claude Paroz <claude@…>):
In [changeset:"df30ae07fc7d7500bbbe51ed3361982f645169f2"]:
{{{
#!CommitTicketReference repository=""
revision="df30ae07fc7d7500bbbe51ed3361982f645169f2"
Fixed postgis test database initialization
Refs #20968. Allow querying template_postgis presence without
existing test database.
Thanks Tim Graham for the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/20968#comment:21>