[Django] #30489: Django RasterField deserialization bug with pixeltype flags
Django
-----------------------------------------+-----------------------------
Reporter: Ivor Bosloper | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 2.2
Severity: Normal | Keywords: RasterField
Triage Stage: Unreviewed | Has patch: 1
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+-----------------------------
After inserting some raster data with raster2pgsql into a Django model
table with a RasterField column, I get a `list index out of range` when
querying the table with a Django Queryset.
{{{
...
File "django/contrib/gis/db/models/fields.py" in from_db_value
360. return connection.ops.parse_raster(value)
File "django/contrib/gis/db/backends/postgis/operations.py" in
parse_raster
369. return from_pgraster(value)
File "django/contrib/gis/db/backends/postgis/pgraster.py" in from_pgraster
57. pixeltype = POSTGIS_TO_GDAL[pixeltype]
}}}
It turns out the `pixeltype` value used is 39 while the `POSTGIS_TO_GDAL`
list is only 16 elements long. The database field contains valid data but
can not be deserialized with Django.
**Steps for reproduction:**
{{{
# Django model
class RasterModel(models.Model):
rast = models.RasterField(srid=4326)
# raw sql, single pixel raster with nodata bit set
insert into app_rastermodel values(1, REPLACE('01 0000 0100
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 E6100000 0100 0100 6 2 03 03', ' ',
'')::raster);
# query generating Exception
RasterModel.objects.get(pk=1)
}}}
**Analysis**: if we look at the
[https://trac.osgeo.org/postgis/wiki/WKTRaster/RFC/RFC1_V0SerialFormat
Raster specification], the pixeltype is a byte of which the 4 highest bits
are flags and the lowest 4 bits are the real pixeltype, but the Django
deserialization code only considers one bit-flag:
{{{
# django/contrib/gis/db/backends/postgis/pgraster.py
def from_pgraster(data):
...
# Subtract nodata byte from band nodata value if it exists
has_nodata = pixeltype >= 64
if has_nodata:
pixeltype -= 64
...
}}}
I have created (my first django) patch and hope somebody can assist me in
getting it correct and merged.
--
Ticket URL: <https://code.djangoproject.com/ticket/30489>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Ivor Bosloper | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 2.2
Keywords: RasterField | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Ivor Bosloper:
Old description:
New description:
**Steps for reproduction:**
The erroneous pixeltype 39 in my example actually had the
`BANDTYPE_FLAG_ISNODATA` (32) bit set which indicates all rastervalues are
nodata.
I have created (my first django) patch and hope somebody can assist me in
getting it correct and merged.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:1>
Django
Reporter: Ivor Bosloper | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 2.2
Keywords: RasterField | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Ivor Bosloper):
See https://github.com/django/django/pull/11381
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:2>
Django
Reporter: Ivor Bosloper | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 2.2
Keywords: RasterField | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:3>
Django
Reporter: Ivor Bosloper | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 2.2
Keywords: RasterField | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Ivor Bosloper:
Old description:
> After inserting some raster data with raster2pgsql into a Django model
> The erroneous pixeltype 39 in my example actually had the
> `BANDTYPE_FLAG_ISNODATA` (32) bit set which indicates all rastervalues
> are nodata.
>
> I have created (my first django) patch and hope somebody can assist me in
> getting it correct and merged.
New description:
**Steps for reproduction:**
are flags and the lowest 4 bits are the real pixeltype. Quoting the
specification:
{{{
Pixel type and storage flag
---------------------------
Pixel type specifies type of pixel values in a band.
Storage flag specifies whether the band data is stored
as part of the datum or is to be found on the server's
filesytem.
There are currently 11 supported pixel value types, so 4
bits are enough to account for all. We'll reserve
the upper 4 bits for generic flags and define upmost as
storage flag:
#define BANDTYPE_FLAGS_MASK 0xF0
#define BANDTYPE_PIXTYPE_MASK 0x0F
#define BANDTYPE_FLAG_OFFDB (1<<7)
#define BANDTYPE_FLAG_HASNODATA (1<<6)
#define BANDTYPE_FLAG_ISNODATA (1<<5)
#define BANDTYPE_FLAG_RESERVED3 (1<<4)
}}}
However, Django deserialization code only considers a single flag
(`BANDTYPE_FLAG_HASNODATA`, bit 6, value 64):
{{{
# django/contrib/gis/db/backends/postgis/pgraster.py
def from_pgraster(data):
...
# Subtract nodata byte from band nodata value if it exists
has_nodata = pixeltype >= 64
if has_nodata:
pixeltype -= 64
...
}}}
The erroneous pixeltype 39 in my example actually had the
`BANDTYPE_FLAG_ISNODATA` (bit 5, value 32) bit set which indicates all
rastervalues are nodata.
I have created (my first django) patch and hope somebody can assist me in
getting it correct and merged.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:4>
Django
Reporter: Ivor Bosloper | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 2.2
Keywords: RasterField | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Daniel Wiesmann):
This is indeed a bug in the raster parser. I guess it has not shown so far
because the flags are not an issue as long as you create rasters using
Django. GDAL (and thuse GDALRaster) does not know about a "all nodata"
flag as far as I know, and it does not allow things like different
datatypes in bands in the same raster.
I had a look at the PR, but I am not familiar with bitwise operators in
python, so I need a little more time. I'll do a more detailed review
tomorrow and share my thoughts on GitHub.
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:5>
Django
Reporter: Ivor Bosloper | Owner: Ivor Bosloper
Type: Bug | Status: assigned
Component: GIS | Version: 2.2
Severity: Normal | Resolution:
Keywords: RasterField | Triage Stage: Accepted
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* owner: nobody => Ivor Bosloper
* needs_better_patch: 0 => 1
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:6>
Django
Reporter: Ivor Bosloper | Owner: Ivor Bosloper
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: RasterField | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Hasan Ramezani):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:7>
Django
Reporter: Ivor Bosloper | Owner: Ivor Bosloper
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: RasterField | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Hasan Ramezani):
* cc: Hasan Ramezani (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:8>
Django
Reporter: Ivor Bosloper | Owner: Ivor Bosloper
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: RasterField | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"828e3b1335e2614d338a630fd5b5f88d38a6b5d2" 828e3b13]:
{{{
#!CommitTicketReference repository=""
revision="828e3b1335e2614d338a630fd5b5f88d38a6b5d2"
Refs #30489 -- Made from_pgraster()/to_pgraster() use
BANDTYPE_FLAG_HASNODATA and bitwise operators for nodata flag.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:9>
Django
Reporter: Ivor Bosloper | Owner: Ivor
| Bosloper
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: RasterField | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* version: 2.2 => master
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:10>
Django
Reporter: Ivor Bosloper | Owner: Ivor
| Bosloper
Component: GIS | Version: master
Severity: Normal | Resolution: fixed
Keywords: RasterField | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"7e15795bf06d362f20257d2e9db378ba8940dc39" 7e15795]:
{{{
#!CommitTicketReference repository=""
revision="7e15795bf06d362f20257d2e9db378ba8940dc39"
Fixed #30489 -- Fixed RasterField deserialization with pixeltype flags.
Thanks Ivor Bosloper for the original patch.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:11>
Django
Reporter: Ivor Bosloper | Owner: Ivor
| Bosloper
Type: Bug | Status: closed
Component: GIS | Version: master
Severity: Normal | Resolution: fixed
Keywords: RasterField | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"9358da704ea9ba55f22df912e47b54eb85d5c97e" 9358da7]:
{{{
#!CommitTicketReference repository=""
revision="9358da704ea9ba55f22df912e47b54eb85d5c97e"
Refs #30489 -- Fixed
RasterFieldTest.test_deserialize_with_pixeltype_flags() when run without
numpy.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:12>