GeoDjango OffdbRasterField

[Piero Toffanin]

Hello,

Not sure this is the right place to post this, if not, could somebody point me to the right place?

I recently had the need to use GeoDjango to define a model that uses a RasterField to store a GeoTIFF raster. https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield

Upon testing, one of the users of the application found that loading a large GeoTIFF (100Mb+) caused PostgreSQL to fail. More details about the report are available here: https://github.com/OpenDroneMap/WebODM/issues/55

I went around the problem by using a relatively new feature of PostGIS that allows raster files to be stored off db. I noticed that RasterField does not support such feature, so I wrote the code to enable support for it via a new OffdbRasterField (https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py). The from_pgraster and to_pgraster functions are modified versions of the same functions found here: https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py

Just wanted to see if there was an interest in adding off db raster support into GeoDjango core. Perhaps by modifying RasterField to have an additional parameter "offdb=True|False" and implement the necessary logic?

Thanks,

-Piero 

[Adam Johnson]

I can't say I'm that familiar with GeoDjango, but that does sound like a useful feature. Where does the data get stored if not in the DB? And does this feature exist on any of the other database backends that GeoDjango supports?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Adam

[Piero Toffanin]

It's stored on the file system. This is to improve performance when storing large geospatial datasets.

This would only work on PostGIS.

On Thursday, December 15, 2016 at 3:11:37 PM UTC-5, Adam Johnson wrote:

I can't say I'm that familiar with GeoDjango, but that does sound like a useful feature. Where does the data get stored if not in the DB? And does this feature exist on any of the other database backends that GeoDjango supports?

On 14 December 2016 at 18:40, Piero Toffanin <pier...@gmail.com> wrote:

Hello,

Not sure this is the right place to post this, if not, could somebody point me to the right place?

I recently had the need to use GeoDjango to define a model that uses a RasterField to store a GeoTIFF raster. https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield

Upon testing, one of the users of the application found that loading a large GeoTIFF (100Mb+) caused PostgreSQL to fail. More details about the report are available here: https://github.com/OpenDroneMap/WebODM/issues/55

I went around the problem by using a relatively new feature of PostGIS that allows raster files to be stored off db. I noticed that RasterField does not support such feature, so I wrote the code to enable support for it via a new OffdbRasterField (https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py). The from_pgraster and to_pgraster functions are modified versions of the same functions found here: https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py

Just wanted to see if there was an interest in adding off db raster support into GeoDjango core. Perhaps by modifying RasterField to have an additional parameter "offdb=True|False" and implement the necessary logic?

Thanks,

-Piero 

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Adam

[Tim Graham]

Daniel Wiesmann did all the work for RasterField. I'm not sure if he follows this list but you can find his email address in the Django commit longs and mail him to ask for his input.

https://github.com/yellowcap

[Daniel Wiesmann]

This is an interesting idea and approach, thanks Piero for the suggestion and the proposed solution.

I agree with Piero and Adam, it would be a great addition to the raster field. Especially for large volumes of data (not only for individual large files, but also for many small files). Raster data can eat up a lot of storage quickly, and delegating that to an off-db storage is useful.

I have been looking at the out-of-db feature for PostGIS rasters, but I am not sure if it can be used in the Django context. The problem I see is the storage location.

If I understand it right, then the raster files need to be on the same system as the PostGIS instance, i.e. they need to be available on the filesystem of the postgis server. Like that, PostGIS can access the files from within its raster operations (such as intersections, getting pixel values etc), and it will work as if the data was stored in the db directly.

In that case however, the out-of-db field would only work if Django is running on the same server as the PostGIS instance. So I think it would fail for more distributed systems, where you have multiple application instances, and a remote PostGIS, which might have replicas etc.

In your experience Piero, is that correct? How are the out-of-db raster files stored by PostGIS?

I have never used the out-of-db raster option so I am not sure if I understood the storage mechanism correclty.

Coincidently, I recently had a similar problem, and I have been testing a version of a raster field which is a subclass of the regular Django FileField. In this approach, the storage can be any Django compatible storage (including remote object storages such as S3). The downside is that PostGIS will no longer recognize the data as rasters, so lookups and PostGIS internal functions will not be available.

This is still quite experimental (it uses a patched Django with an extension of the GDALRaster) and is slightly hacky, but does convert the remote rasters into GDALRaster instances when opened.

https://github.com/geodesign/django-raster/compare/raster_file_field

An ideal solution would be a mixture of the two, but I am not sure if PostGIS can handle remote storages.

[Piero Toffanin]

Your analysis is spot on, PostGIS references the path to off-db rasters using an absolute path, which needs to be resolved on the PostGIS server.

I guess a developer could map a folder (via samba for example) on the PostGIS server to match that of the Django server so that both servers share the same path structure. There would probably be some problems with this approach as well, especially when mixing different operating systems, or synchronization issues depending on the type of network mapping used.

If this addition is too problematic to be added in core, I could just isolate the code and release it as a plugin, with some documentation about the limits of the approach.

[Daniel Wiesmann]

I am not sure what the policy is for components with such strong
restrictions. So I guess this is a question back to the group:

What are requirements for components within Django with regard to system
architecture? Are components required to work on distributed systems?

If there is no general answer to this question, please advise on this
specific case.

> <https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py>).

> The from_pgraster and to_pgraster functions are
> modified versions of the same functions found here:
> https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py
> <https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py>
>
> Just wanted to see if there was an interest in
> adding off db raster support into GeoDjango core.
> Perhaps by modifying RasterField to have an
> additional parameter "offdb=True|False" and
> implement the necessary logic?
>
> Thanks,
>
> -Piero
>
> --
> You received this message because you are subscribed
> to the Google Groups "Django developers
> (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving
> emails from it, send an email to
> django-develop...@googlegroups.com.
> To post to this group, send email to
> django-d...@googlegroups.com.
> Visit this group at
> https://groups.google.com/group/django-developers

> <https://groups.google.com/group/django-developers>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com

> <https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to a topic in the

> Google Groups "Django developers (Contributions to Django itself)" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/Drb2qYuuuKo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> django-develop...@googlegroups.com
> <mailto:django-develop...@googlegroups.com>.

> To post to this group, send email to django-d...@googlegroups.com

> <mailto:django-d...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/28150c14-089d-4025-b52b-f616fb0a600e%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/28150c14-089d-4025-b52b-f616fb0a600e%40googlegroups.com?utm_medium=email&utm_source=footer>.