Adding a RasterField makes sense as a first step to handling raster data
input and output. Spatial querying and other operations could then be
built on top of that.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
* needs_better_patch: => 0
* needs_docs: => 0
* needs_tests: => 0
* stage: Unreviewed => Accepted
Comment:
(Daniel is author of https://github.com/geodesign/django-raster/)
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:1>
* owner: nobody => yellowcap
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:2>
Comment (by yellowcap):
I have made good process in creating some basic raster functionality,
including gdal bindings, a GDALRaster object, a RasterField and a Tiler
engine to create tiles from GDALRasters, and tests for most of it.
For now I have kept all new code in a submodule of contrib/gis/gdal, see
https://github.com/geodesign/django/tree/gdalraster/django/contrib/gis/gdal/raster
Now I was thinking of moving the module to master and starting to write
the documentation for it. But I am not sure if it would be better to
integrate it directly into the contrib.gis.gdal module. In the gdal.raster
module, I have mimicked the structure of that module (for instance by
creating a prototypes/ds.py file with the ctypes gdal bindings).
Does anyone have time to give me some advice or discuss how to proceed
from here?
Note: I branched off the ''stable/1.7.x'' branch because like this I am
able to use the code in one of my projects and play with it in a full
project development environment.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:3>
Comment (by claudep):
Nice to see quick progress!
I think that we could aim for more integration with main gdal module. For
example, when I compare driver.py and raster/driver.py, I see much code
duplication. The `RasterField` model should live somewhere in
`gis/db/models` (maybe by creating a fields module). I also would like to
see all prototypes in gdal/prototypes and tests in gdal/tests. Maybe the
other files could be left in a `raster` module. These are just quick
thoughts, without deep looking in the code.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:4>
Comment (by claudep):
Daniel, look at this pull request:
https://github.com/django/django/pull/3648
Could you review/comment on it, and tell if you think it's going in a good
direction?
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:5>
Comment (by Claude Paroz <claude@…>):
In [changeset:"4df3a3e0e9bfcd64a3e2eb85e0d4449d7b8f4f6e"]:
{{{
#!CommitTicketReference repository=""
revision="4df3a3e0e9bfcd64a3e2eb85e0d4449d7b8f4f6e"
Added GDAL prototypes for raster support
Refs #23804.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:6>
Comment (by Claude Paroz <claude@…>):
In [changeset:"00fa1474d7d2dbd5530a650ea9fe09f36bff77c7"]:
{{{
#!CommitTicketReference repository=""
revision="00fa1474d7d2dbd5530a650ea9fe09f36bff77c7"
Added raster support for GDAL Driver class
Based on Daniel Wiesmann's work. Refs #23804.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:7>
Comment (by claudep):
The base is now in. I'll continue to work on integrating Daniel's branch.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:8>
Comment (by yellowcap):
I branched off master and started integrating the raster files into the
structure with the new driver. I followed your suggestions above. The new
branch is this one:
https://github.com/geodesign/django/tree/raster/django/contrib/gis/gdal
- Moved all tests into gdal.tests folder
- Separated constants from utils functions
- Moved RasterField into `django.contrib.gis.db.models.fields` module
Most of the changes are file-rearangements.
The raster field can leverage some of the features around the
GeometryFields, but here some of the code that can be re-used is named
Geometry* for instance the GeometryProxy and the `geom_type` property.
Both are related to lazy-instanciation of field target instances (Rasters
or Geometries).
See:
https://github.com/geodesign/django/blob/raster/django/contrib/gis/db/models/fields.py#L356
https://github.com/geodesign/django/blob/raster/django/contrib/gis/db/models/fields.py#L390
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:9>
Comment (by yellowcap):
The RasterField is still very minimalistic, here are some thoughts of what
is missing/could be improved:
- Integration with different db backends. Currently the raster IO is
tailored towrards PostGIS. The value passed to the db is the output of the
`wkb` property of the GDALRaster, which converts between PostGIS WKB and
GDALRaster. This seems to work in the test-databases. The db_type is fixed
to `'raster'`, which is also PostGIS specific.
- A deeper integration with the GeometryField would be beneficial. For
instance the SRID property could be shared in a master class. But this is
currently integrated into GeometryField directly. So integrating the two
would probably require a GeoField master class or so.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:10>
Comment (by claudep):
I've prepared a new (small) step, with the following patch. I've
deliberately chosen some different paths from your branch. It's not to
contradict you by pleasure :-), I may totally be wrong or naive with
certain choices, but I'm trying to create the most Pythonesque API as
possible. Feel free to criticize and contradict me in return!
https://github.com/django/django/pull/3694
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:11>
Comment (by yellowcap):
Just had a quick look, this is awsome, its very clean like this. I can't
resist to add a quote of a movie I saw the other day:
"You've got to understand your limitations. It's your limitations that
make you the wonderful disaster that you most probably are. For me that is
where collaboration comes in. To take an idea that is blind and unformed
and that has been hatched largely in solitude and allow these strange
collaborator creatures to morph it into something else, something that is
better, that's really something to see."
-- Nick Cave on collaboration from 20,000 Days on Earth
My main concern for now on this is that it limits the data source to be
only from a file. You can't instantiate a source from memory, which is
what I had intended to use in the raster field. And in-memory rasters work
just the same way as the file based ones.
Were you thinking of having a separate GDALRaster object that can be
created from memory and used in a field? If you have time, maybe you could
briefly summarize how you would see the rest of the code to look like.
Will have a closer look over the next few days.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:12>
Comment (by claudep):
Thanks for the quote :-)
> My main concern for now on this is that it limits the data source to be
only from a file.
Don't fear, it's just a technique, going step after step. Of course, the
current functionality is almost unusable for anything serious. And like
you I'm looking forward to new possibilities. Don't stop to write code, I
wouldn't be able to write anything without taking inspiration from your
work, seriously.
Now I may at some point question the extent of the code needed to be in
the framework and the code which could be better in an external project.
For me the `RasterField` code is clearly an objective at short term. But
the tiling functionality, for example, I'm still not completely convinced
it belongs to Django core. We'll see.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:13>
Comment (by yellowcap):
I am aware that some of the code in the raster branch might be too high
level for Django core and am happy to discuss this. Here is a start:
''For the img property''
I kept the `img` property as part of it as this might be needed for
showing the raster in the admin interface.
''For the Tiler egine/utility''
I envisioned the Tiler engine to be the raster version of the LayerMapping
utility (see link below). For larger rasters, it is unpractical to load
them as one single raster, so in many use cases some sort of tiling will
be necessary. The tiler I wrote is very focused on creating tiles for TMS,
which is probably not general enough as a tiling utitlity. Still I think
something along those lines is necessary for loading large rasters from
files into a db.
[https://github.com/django/django/blob/master/django/contrib/gis/utils/layermapping.py]
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:14>
Comment (by Tim Graham <timograham@…>):
In [changeset:"f3eed95175f8a8a365218a09fdc2dc864ff75d0c"]:
{{{
#!CommitTicketReference repository=""
revision="f3eed95175f8a8a365218a09fdc2dc864ff75d0c"
Removed netCDF from GIS driver testing; refs #23804.
It may not be installed on all systems.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:15>
Comment (by Claude Paroz <claude@…>):
In [changeset:"6e08bde8c4525dda7d82bbf55b4b45a6e16213da"]:
{{{
#!CommitTicketReference repository=""
revision="6e08bde8c4525dda7d82bbf55b4b45a6e16213da"
Added RasterSource/GDALBand GDAL objects
Based on Daniel Wiesmann's raster branch. Thanks Daniel Wiesmann
and Tim Graham for the reviews. Refs #23804.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:16>
Comment (by claudep):
Note that to add the RasterField, we'll probably be facing the same issue
as #23879 (skipping model creation for non PostGIS).
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:17>
Comment (by yellowcap):
I was working on integrating pixel value read and write support. When you
have time, please have a look at the following pull request:
https://github.com/django/django/pull/3909
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:18>
Comment (by claudep):
Thanks for the great work, I will definitely put that in my review list.
Unfortunately, this will not be included in Django 1.8 (currently feature
frozen), but let's aim for a bright raster support in Django 1.9
(including the raster db field, hopefully)!
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:19>
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:20>
Comment (by yellowcap):
I added a HEX representation for the GDAL Rasters to the above pull
request. Rasters can be instantiated from HEX data and converted into that
format too.
I was trying to find a standard for Raster data in binary/hex but
unfortunately such a standard does not seem to exist.
http://gis.stackexchange.com/questions/130209/is-there-an-open-standard-
for-raster-information
http://lists.osgeo.org/pipermail/gdal-dev/2015-January/040807.html
So in my opinion the PostGIS raster HEX representation recommends itself
here as it is well defined and is the native format of rasters in PostGIS.
The implementation in my pull request uses the PostGIS Raster hex
definition.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:21>
Comment (by yellowcap):
I added a first take at a RasterField for PostGIS backends to the pull
request.
The testing is very basic and I have not yet written any documentation for
it and it lacks proper Index creation. However, I wanted to get feedback
on the basic implementation before workin on it more.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:22>
Comment (by claudep):
Sorry, I was busy with #24214 which is no1 priority for GIS in 1.9. I
should have time to review part of your work soon.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:23>
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:24>
Comment (by Claude Paroz <claude@…>):
In [changeset:"f269c1d6f6dcc22c0a781f3223c6da0a4483b06e" f269c1d6]:
{{{
#!CommitTicketReference repository=""
revision="f269c1d6f6dcc22c0a781f3223c6da0a4483b06e"
Added write support for GDALRaster
- Instantiation of GDALRaster instances from dict or json data.
- Retrieve and write pixel values in GDALBand objects.
- Support for the GDALFlushCache in gdal C prototypes
- Added private flush method to GDALRaster to make sure all
data is written to files when file-based rasters are changed.
- Replaced ``ptr`` with ``_ptr`` for internal ptr variable
Refs #23804. Thanks Claude Paroz and Tim Graham for the reviews.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:25>
Comment (by Tim Graham <timograham@…>):
In [changeset:"b9cb81570e24284156e09ab1fdc3c19a8d563b07" b9cb815]:
{{{
#!CommitTicketReference repository=""
revision="b9cb81570e24284156e09ab1fdc3c19a8d563b07"
Made SRID a required parameter for GDALRaster instantiation; refs #23804.
Earlier versions of GDAL do not allow the srid to be set to 0,
so it should be a required parameter to ensure compatibility.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:26>
Comment (by yellowcap):
Here is a first draft of a Raster field for PostGIS >= 2.0 backends. I am
not sure if I covered all the important aspects of creating a field, but
it should serve as a starting point at least.
https://github.com/django/django/pull/4339
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:27>
Comment (by Tim Graham <timograham@…>):
In [changeset:"0d9b018e07c384314e142372153eb670c2e129f3" 0d9b018e]:
{{{
#!CommitTicketReference repository=""
revision="0d9b018e07c384314e142372153eb670c2e129f3"
Fixed gis test failures when numpy isn't installed.
Thanks to Bas Peschier for pointing this out. Refs #23804.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:28>
Comment (by yellowcap):
I updated the pull request for the first RasterField implementation. The
raster field should now be fairly complete, but it became quite large due
to that. Its hard to split things at this point though, as the different
elements are quite interrelated. I hope its still an acceptable size.
Here is a description of the current state:
- The field is implemented for PostGIS backends only.
- Similar to `GeometryFields`, the field srid can be specified and the
raster will automatically be transformed to the field srid upon saving.
- A spatial index for the raster is built by default, and can be disabled
with `spatial_index=False`.
- The `null` and `blank` flags can be specified.
- Lazy instantiation works for the raster field as well.
- Tests are in place for the RasterField and corresponding migration
commands and operations
For this, I made some changes on the existing codebase:
- Generalized the `GeometryProxy` and renamed to `SpatialProxy`. This is
used for lazy-instantiation of objects related to spatial fields.
- Override the `get_indexes` introspection function for postgis, as the
spatial indices are based on expressions and are not straight indices
(using `ST_ConvexHull`). So the sql code retrieving the list of indices
has to be adopted to that.
- Update the `get_placeholder` function in the `PostGISOperator` to
implicitly reproject rasters that are passed to a field with a different
srid than the field srid.
- Change in `PostGISSchemaEditor` to create spatial indices on raster
(wrapping the index creation sql with the `ST_ConvexHull` function).
- Split parts of the `GeometryField` into a generalized `BaseSpatialField`
class. The `GeometryField` and the `RasterField` subclass the
`BaseSpatialField`.
- Updated the `gis_migrations` test module to also include the
`RasterField` into the tests.
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:29>
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:30>
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"b769bbd4f6a3cd1bcd9ebf3559ec6ea0f9b50565" b769bbd4]:
{{{
#!CommitTicketReference repository=""
revision="b769bbd4f6a3cd1bcd9ebf3559ec6ea0f9b50565"
Fixed #23804 -- Added RasterField for PostGIS.
Thanks to Tim Graham and Claude Paroz for the reviews and patches.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:31>
Comment (by Tim Graham <timograham@…>):
In [changeset:"c0fff64486198ee8885f4f951399a3675f01e3cd" c0fff64]:
{{{
#!CommitTicketReference repository=""
revision="c0fff64486198ee8885f4f951399a3675f01e3cd"
Fixed #25011, Refs #23804 -- Added check for GDAL on RasterField
initialization
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:32>
Comment (by Tim Graham <timograham@…>):
In [changeset:"9509268cea9ed1dfda5e66a25f4429a25bf40480" 9509268]:
{{{
#!CommitTicketReference repository=""
revision="9509268cea9ed1dfda5e66a25f4429a25bf40480"
Refs #23804 -- Improved value validation in GDALRaster.geotransform
setter.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23804#comment:33>