[Django] #37179: Slicing a GDAL layer should scan layer once only

[Django]

#37179: Slicing a GDAL layer should scan layer once only
------------------------------------------------+--------------------------
Reporter: wongcht | Owner: wongcht
Type: Cleanup/optimization | Status: assigned
Component: GIS | Version: dev
Severity: Normal | Keywords: gdal
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 1
UI/UX: 0 |
------------------------------------------------+--------------------------
In `django.contrib.gis.gdal.layer`'s `Layer.__getitem__`, if a layer (with
**n** features) **does not support random read**, slicing the layer would
call `_make_feature` at most **n** times which this method scan the whole
layer until feature id matched.

E.g. Slicing a layer for 3 features (id= 1,2,3) would scan the layer 3
times: [1] for id=1, [1,2] for id=2, [1,2,3] for id=3. Worse case would
call `__iter__` (`capi.get_next_feature`) for **n(n+1)/2** times

A better iteration should scan the layer once only to get all features
required. A possible [https://github.com/wongcht/django/pull/2/changes
fix] FYR
--
Ticket URL: <https://code.djangoproject.com/ticket/37179>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37179: Slicing a GDAL layer should scan layer once only

-------------------------------------+-------------------------------------
Reporter: wongcht | Owner: wongcht
Type: | Status: assigned
Cleanup/optimization |
Component: GIS | Version: dev
Severity: Normal | Resolution:

Keywords: gdal | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0

-------------------------------------+-------------------------------------
Description changed by wongcht:

Old description:

> In `django.contrib.gis.gdal.layer`'s `Layer.__getitem__`, if a layer
> (with **n** features) **does not support random read**, slicing the layer
> would call `_make_feature` at most **n** times which this method scan the
> whole layer until feature id matched.
>
> E.g. Slicing a layer for 3 features (id= 1,2,3) would scan the layer 3
> times: [1] for id=1, [1,2] for id=2, [1,2,3] for id=3. Worse case would
> call `__iter__` (`capi.get_next_feature`) for **n(n+1)/2** times
>
> A better iteration should scan the layer once only to get all features
> required. A possible [https://github.com/wongcht/django/pull/2/changes
> fix] FYR

New description:

In `django.contrib.gis.gdal.layer`'s `Layer.__getitem__`, if a layer (with
**n** features) **does not support random read**, slicing the layer would
call `_make_feature` at most **n** times which this method scan the whole
layer until feature id matched.

E.g. Slicing a layer for 3 features (id= 1,2,3) would scan the layer 3

times: [1] for id=1, [1,2] for id=2, [1,2,3] for id=3. Worst case would

call `__iter__` (`capi.get_next_feature`) for **n(n+1)/2** times

A better iteration should scan the layer once only to get all features
required. A possible [https://github.com/wongcht/django/pull/2/changes
fix] FYR

--

--
Ticket URL: <https://code.djangoproject.com/ticket/37179#comment:1>

[Django]

#37179: Slicing a GDAL layer should scan layer once only
-------------------------------------+-------------------------------------
Reporter: wongcht | Owner: wongcht

Type: | Status: closed

Cleanup/optimization |
Component: GIS | Version: dev

Severity: Normal | Resolution: needsinfo

Keywords: gdal | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by David Smith):

* resolution: => needsinfo
* status: assigned => closed

Comment:

A benchmark script for performance improvements is helpful to assess both
the current situation and to compare any improvements against.

I wonder if a simplification of the current code is enough.
[https://gdal.org/en/latest/doxygen/ogr__api_8h.html#a29d378c5092db944966398de8d1ac964
get_featre()] has a fallback if random read is not supported and therefore
we could rely on the underlying c library. Reducing the number of c api
calls is likely to be beneficial, but would need to have a benchmark to
prove.

That is `_make_feature()` could become.

{{{
def _make_feature(self, feat_id):
"""
Helper routine for __getitem__ that constructs a Feature from the
given
Feature ID.
"""
try:
return Feature(capi.get_feature(self.ptr, feat_id), self)
except GDALException:
pass
raise IndexError("Invalid feature id: %s." % feat_id)
}}}


I'll mark as needs info - a benchmark script showing the performance issue
would be the next step here. Do you have the capacity to write one up?
--
Ticket URL: <https://code.djangoproject.com/ticket/37179#comment:2>