[Django] #29989: Geodjango running through memory
Django
-------------------------------------+-------------------------------------
Reporter: tjquinn1 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 2.1
Severity: Normal | Keywords: raster, geodjango,
Triage Stage: | postgis
Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I have been trying to upload a raser to my postgis backend and keep
running into
{{{#!python
django.db.utils.OperationalError: cannot allocate memory for output buffer
}}}
I try to upload the file in the django shell using the following:
{{{#!python
raster = GDALRaster('rasters/data/my_image.tif', write=True)
r = RastModel(name="Foo", image=raster)
r.save()
}}}
The image is only about a gb and I have plenty of ram to handle it. When I
save the raster it runs through almost all 32gb of ram and then throws the
error. From all the docs and posts I have seen it appears that I am
uploading the raster correctly so I am assuming there is a bug somewhere.
--
Ticket URL: <https://code.djangoproject.com/ticket/29989>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: GIS | Version: 2.1
Keywords: raster, geodjango, | Triage Stage:
postgis | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
Please provide a full traceback so we can determine in which phase the
process runs out of memory. I assume it's happening at on the `save()`
call?
Also while I'm not that familiar with GDAL rasters I know that TIFF images
can be heavily compressed so it's possible that even if the ''compressed''
file is only 1GB it could be way larger uncompressed.
--
Ticket URL: <https://code.djangoproject.com/ticket/29989#comment:1>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: GIS | Version: 2.1
Keywords: raster, geodjango, | Triage Stage:
postgis | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> I have been trying to upload a raser to my postgis backend and keep
> running into
> {{{#!python
> django.db.utils.OperationalError: cannot allocate memory for output
> buffer
> }}}
> I try to upload the file in the django shell using the following:
>
> {{{#!python
> raster = GDALRaster('rasters/data/my_image.tif', write=True)
> r = RastModel(name="Foo", image=raster)
> r.save()
> }}}
> The image is only about a gb and I have plenty of ram to handle it. When
> I save the raster it runs through almost all 32gb of ram and then throws
> the error. From all the docs and posts I have seen it appears that I am
> uploading the raster correctly so I am assuming there is a bug somewhere.
New description:
I have been trying to upload a raser to my postgis backend and keep
running into
{{{#!python
django.db.utils.OperationalError: cannot allocate memory for output buffer
}}}
I try to upload the file in the django shell using the following:
{{{#!python
raster = GDALRaster('rasters/data/my_image.tif', write=True)
r = RastModel(name="Foo", image=raster)
r.save()
}}}
The image is only about a gb and I have plenty of ram to handle it. When I
save the raster it runs through almost all 32gb of ram and then throws the
error. From all the docs and posts I have seen it appears that I am
uploading the raster correctly so I am assuming there is a bug somewhere.
{{{
>>> rast = GDALRaster('rasters/data/raw_tempe.tif', write=True)
>>> r = Rast(name='Test', image=rast)
>>> r.save()
Traceback (most recent call last):
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
psycopg2.OperationalError: cannot allocate memory for output buffer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/models/base.py", line 718, in save
force_update=force_update, update_fields=update_fields)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/models/base.py", line 748, in save_base
updated = self._save_table(raw, cls, force_insert, force_update,
using, update_fields)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/models/base.py", line 831, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk,
raw)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/models/base.py", line 869, in _do_insert
using=using, raw=raw)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/models/query.py", line 1136, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/models/sql/compiler.py", line 1289, in execute_sql
cursor.execute(sql, params)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False,
executor=self._execute)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/mnt/f/Projects/gis322/env/lib/python3.6/site-
packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
django.db.utils.OperationalError: cannot allocate memory for output buffer
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/29989#comment:2>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: GIS | Version: 2.1
Keywords: raster, geodjango, | Triage Stage:
postgis | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Taylor):
Replying to [comment:1 Simon Charette]:
> Please provide a full traceback so we can determine in which phase the
process runs out of memory. I assume it's happening at on the `save()`
call?
>
> Also while I'm not that familiar with GDAL rasters I know that TIFF
images can be heavily compressed so it's possible that even if the
''compressed'' file is only 1GB it could be way larger uncompressed.
I added the traceback above let me know if I can supply anything else to
help resolve the issue. If wanted I could supply the image as well since
it is freely available landsat imagery.
--
Ticket URL: <https://code.djangoproject.com/ticket/29989#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: GIS | Version: 2.1
Keywords: raster, geodjango, | Triage Stage:
postgis | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> I have been trying to upload a raser to my postgis backend and keep
> running into
> {{{#!python
> django.db.utils.OperationalError: cannot allocate memory for output
> buffer
> }}}
> I try to upload the file in the django shell using the following:
>
> {{{#!python
> raster = GDALRaster('rasters/data/my_image.tif', write=True)
> r = RastModel(name="Foo", image=raster)
> r.save()
> }}}
>
> The image is only about a gb and I have plenty of ram to handle it. When
> I save the raster it runs through almost all 32gb of ram and then throws
> the error. From all the docs and posts I have seen it appears that I am
> uploading the raster correctly so I am assuming there is a bug somewhere.
>
> django.db.utils.OperationalError: cannot allocate memory for output
> buffer
>
> }}}
New description:
I have been trying to upload a raster to my postgis backend and keep
running into
{{{#!python
django.db.utils.OperationalError: cannot allocate memory for output buffer
}}}
I try to upload the file in the django shell using the following:
{{{#!python
raster = GDALRaster('rasters/data/my_image.tif', write=True)
r = RastModel(name="Foo", image=raster)
r.save()
}}}
The image is only about a gb and I have plenty of ram to handle it. When I
save the raster it runs through almost all 32gb of ram and then throws the
error. From all the docs and posts I have seen it appears that I am
uploading the raster correctly so I am assuming there is a bug somewhere.
{{{
django.db.utils.OperationalError: cannot allocate memory for output buffer
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/29989#comment:4>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: GIS | Version: 2.1
Severity: Normal | Resolution: invalid
Keywords: raster, geodjango, | Triage Stage:
postgis | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => invalid
Comment:
Yes, when opening a ticket you should include enough details to reproduce
the problem.
If you can't debug the issue yourself to explain why Django is at fault,
generally, you should ask for help on
[wiki:TicketClosingReasons/UseSupportChannels our support channels] before
opening a ticket. I see you already opened
[https://stackoverflow.com/questions/53475849/uploading-large-raster-
geodjango a question on Stackoverflow] so that's fine. Please don't use
this ticket tracker as a second level support channel.
I see that the exception comes from PostgreSQL, not Django. If you find
that Django is at fault, feel free to reopen.
--
Ticket URL: <https://code.djangoproject.com/ticket/29989#comment:5>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: GIS | Version: 2.1
Severity: Normal | Resolution: invalid
Keywords: raster, geodjango, | Triage Stage:
postgis | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
Taylor, you might want to try getting help from the
[https://groups.google.com/forum/#!forum/geodjango geodjango mailing list
as well] or the [https://gis.stackexchange.com GIS dedicated stack
exchange] instead or the generic StackOverflow.
If you've noticed the process chew through all of your RAM then the
uncompression of the image looks like a smoking gun to me. You could try
using an image manipulation tool to uncompress the image to disk and
compare its file size.
As Tim said nothing proves this is an actual bug with Django so we cannot
dedicate time to investigate further. If you can come up with a more
detailed report of what is happening please report it back here even if
Django is not at fault as it might help others hitting a similar issue.
Finally I'd suggest you include your exact model definition in your
reports as well as it's not completely clear whether or not
`RastModel.image` is an `ImageField` or `RasterField` and mention your are
using PostGIS with a `RASTER` column on the GIS stack exchange in the
latter case. Best of luck.
--
Ticket URL: <https://code.djangoproject.com/ticket/29989#comment:6>