Hypatia - drop Python 2, GitHub Actions?

[Peter Wilkinson]

Hi,

I’ve been a user of Hypatia for a fair while now and have an additional index type I’m just cleaning up at the moment to propose in a pull request - it’s a spatial index based on a port of RBush (https://github.com/mourner/rbush) from JavaScript. As part of this I’ve been testing with all supported Python 3 versions and 3.12 alpha versions as I’d like to be able to take advantage of some of the Python 3 additions and the code it is used in is only Python 3. I’ve also made some changes to allow for running the tox tests via GitHub actions rather than using Travis CI - it’s a little simpler with one less moving part and I’ve had recent experience with Actions (https://github.com/pfw/hypatia/actions/runs/3722816192) which might be useful. I’d like to get the docs building and published as well.

How would people feel about dropping support for anything below 3.7? Given there is a now a 0.4 version that is on PyPI I don’t think there is much loss if future additions only work on Python3.

Is there any appeal in using GitHub Actions for the tests?

Regards,

Peter Wilkinson

[Jonathan Vanasco]

I can't speak for the maintainers, but the Pylons Projects have been standardizing onto Github Actions so there is appeal.  IIRC, support for projects has also been targeting 3.7+.

But as a personal preference, while I don't use this library myself, whenever I move a project onto GitHub actions I initially maintain support for 2.7 and 3.6 then drop those versions in a second commit. The reason is that many people still run 2.7/3.6 in production, so if a security fix is released for those versions the fix can be applied to a commit with a functional test environment.  I find it much easier to just spend an extra 5-10 minutes setting up that environment in advance "just in case" while I'm already working on GitHub Actions, than spending 1-2 hours to suddenly re-familiarize myself with the CI environment/setup and trying to jam legacy Python support into it.

[Bert JW Regeer]

Most of the projects that Michael and I maintain we use the current supported version of Python and up. We drop older versions when we get around to touching the repo because maintaining older versions of Python and testing/validating is a pain.

If you want to put together a PR that moves to Github actions (feel free to steal from any of our other repos, and in fact I encourage it so that it is easier to maintain for us moving forward see; pyramid/waitress/webob/others).

I’m happy to review such a PR.

Python 2 is dead. I would not recommend putting time or effort into it unless someone is paying you a lot of money.

Bert JW Regeer

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/535F0F0A-D352-4E97-9414-D358B361DB5C%40thirdfloor.com.au.

[Peter Wilkinson]

On 20 Dec 2022, at 6:54 am, Jonathan Vanasco <jvan...@gmail.com> wrote:

But as a personal preference, while I don't use this library myself, whenever I move a project onto GitHub actions I initially maintain support for 2.7 and 3.6 then drop those versions in a second commit. The reason is that many people still run 2.7/3.6 in production, so if a security fix is released for those versions the fix can be applied to a commit with a functional test environment.  I find it much easier to just spend an extra 5-10 minutes setting up that environment in advance "just in case" while I'm already working on GitHub Actions, than spending 1-2 hours to suddenly re-familiarize myself with the CI environment/setup and trying to jam legacy Python support into it.

I’ll have a look at that and in general I’d agree but in this specific case since there is little movement in the project over time I don’t think it’s likely to be any updates for just security fixes - if there are security issues they’ve been there for a long time. There is a little bit of traction with some small changes in the last little while and since I use this library quite a bit I’d like to see if it can move forward a bit.

Peter W.

[Peter Wilkinson]

On 20 Dec 2022, at 7:38 am, Bert JW Regeer <xist...@0x58.com> wrote:

If you want to put together a PR that moves to Github actions (feel free to steal from any of our other repos, and in fact I encourage it so that it is easier to maintain for us moving forward see; pyramid/waitress/webob/others).

I’m happy to review such a PR.

Thanks, I’ll get that through shortly, I’ve got it working in my fork and it’s not a massive change.

Python 2 is dead. I would not recommend putting time or effort into it unless someone is paying you a lot of money.

Pretty much my thinking as well - definitely not something I want to spend time on.

Peter W.

[Peter Wilkinson]

On 20 Dec 2022, at 7:38 am, Bert JW Regeer <xist...@0x58.com> wrote:

I’m happy to review such a PR.

I’ve put a PR at https://github.com/Pylons/hypatia/pull/16 which is a relatively small first step to start bringing things inline with Pyramid - I’d like to add listing and formatting as well as remove all of the Python 2 compatibility code but that can wait.

I’d also like to remove the ‘alpha’ notice in README.rst - what is there works very well as it is and from my own experience the ‘alpha’ put me off at the start, although I think the audience is probably limited I don’t think anyone should be afraid to use Hypatia.

Peter W.

[Thierry Florac]

Hi Peter,

I'm also using Hypatia for several years now without any problem, it works very well!

So this "alpha" notice is probably useless!

I'm also interested in using spatial indexes in Hypatia, so I'm quite excited to use your indexes.

And if you need any help to test these features, just ask!  ;)

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

--

You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/A3B67487-5489-4D15-ACFD-80F0759A41C1%40thirdfloor.com.au.

[Peter Wilkinson]

On 21 Dec 2022, at 7:21 pm, Thierry Florac <tfl...@gmail.com> wrote:

I'm also interested in using spatial indexes in Hypatia, so I'm quite excited to use your indexes.

And if you need any help to test these features, just ask!  ;)

Hi Thierry,

Out of interest, what spatial searches are you looking to use? I’ve got straight up intersection on exact geometries (Polygon, MultiPolygon, Points) working and it’s pretty straightforward to support additional predicates, eg. Contains, touches etc. but rather than just dump all in I’d like to get actual use cases that I can write tests for as well.

Peter W.

[Thierry Florac]

Hi Peter,

My main usage are probably :

 - to find objects which are located below a given distance from a reference points

 - to find objects which are contained into a given geometry or overlapping it.

I generally use PostGIS or Elasticsearch to handle these queries, but in some contexts it could be useful to mix these queries with other attributes queries directly from the catalog...

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CE37D3F7-C7F3-4ACD-80FB-CD3C8EC9BE94%40thirdfloor.com.au.

[Peter Wilkinson]

On 22 Dec 2022, at 6:44 pm, Thierry Florac <tfl...@gmail.com> wrote:

My main usage are probably :

 - to find objects which are located below a given distance from a reference points

 - to find objects which are contained into a given geometry or overlapping it.

I generally use PostGIS or Elasticsearch to handle these queries, but in some contexts it could be useful to mix these queries with other attributes queries directly from the catalog…

Hi Thierry,

I’m in a similar position and really don’t want to carry around the complexity of PostGIS installs and syncing data between two systems. I’ve been able to get sub ms results when searching for a point in 300,000+ polygons from the all country admin region dataset at https://gadm.org/download_world.html - that is plenty fast enough for my needs...

At the moment there is code in a fork installable via:

pip install 'git+https://github.com/pfw/hypatia.git@spatial-index#egg=hypatia[spatial]'

Which so far passes all my tests.

It exposes intersects as a Comparator so you can do things like:

from shapely import box

spatialindex.intersects(box(151.20699218799413, -33.864564753236095, 151.20729952635648, -33.86439938579896)).execute().all()

Or combine as you would normally via:

(spatialindex.intersects(box(151.20699218799413, -33.864564753236095, 151.20729952635648, -33.86439938579896)) & otherindex.eq(263)).execute().all()

It does support other predicates than ‘intersects’ but doesn’t expose them as Comparators yet - that will come shortly - have a look at test_belgium.

The underlying data structure can support a KNN query for finding ’nearest to x’ and I’ll add that as well.

It would be great to hear if the current state works for your needs and once the few extra bits are there I’ll clean it up for a PR on the main repo.

Peter W.

[Thierry Florac]

Hi Peter!

I'm not sure to have time to test before next year!  :/

Can you just specify the "format" you use to store geometries?

Maybe it's documented in your package but I didn't have any time to check for it...

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/5EFF8574-F3EF-4DCA-A1DD-FBC95CFBD438%40thirdfloor.com.au.

[Peter Wilkinson]

On 23 Dec 2022, at 11:33 pm, Thierry Florac <tfl...@gmail.com> wrote:

I'm not sure to have time to test before next year!  :/

Can you just specify the "format" you use to store geometries?

Maybe it's documented in your package but I didn't have any time to check for it…

Hi Thierry,

The index requires a shapely (https://shapely.readthedocs.io/) geometry from which it calculates a bounding box  The - having the full geometry allows for it to do an exact intersection. Any of the https://shapely.readthedocs.io/en/stable/geometry.html#geometry-types should work.

I’ve got a very simple demo (below) which takes in data from GADM (https://gadm.org/download_country.html) and there is also some more code in tests for some simplified Belgium geometries.

I’m going to write up some more docs to go along with this in the repo but hopefully this gives you enough to get going after Christmas and the new year.

If this is getting a bit into the weeds for others on a more general pylons list, say the word and Thierry you can ping me directly.

Peter W.

from BTrees.IOBTree import BTree

from hypatia.spatial import SpatialIndex

from persistent import Persistent

from shapely.geometry.base import BaseGeometry

from hypatia.catalog import Catalog

@dataclass(order=True, eq=True)

class Region(Persistent):

    fid: int

    name: str

    geometry: BaseGeometry = field(compare=False)

@dataclass

class Regions(Persistent):

    catalog: Catalog = field(default_factory=Catalog)

    container: BTree = field(default_factory=BTree)

    def __post_init__(self):

        self.catalog["geometry"] = SpatialIndex("geometry")

    def add(self, fid, name, geometry):

        region = Region(fid, name, geometry)

        self.container[fid] = region

        self.catalog.index_doc(fid, region)

And load data with something like (this pulls in 350k + geometries from the whole world download):

from shapely import shape

import fiona

regions = Regions()

for v in filter(None, fiona.open('gadm_410.gpkg')):

regions.add(int(v['id']), v['properties']['NAME_1'], shape(v['geometry']))

From there you can search against any Shapely geometry something like:

paris = shapely.geometry.Point(2.349014, 48.864716)

near_paris = paris.buffer(0.1) 

list(regions.catalog['geometry'].intersects(near_paris).execute(resolver=regions.container.get).all())

Which returns 81 areas like this and takes about 1.5ms on my laptop against the 356508 geometries imported above:

[Region(fid=76354, name='Île-de-France', geometry=<MULTIPOLYGON (((2.275 48.741, 2.284 48.748, 2.276 48.757, 2.288 48.761, 2.2...>),

 Region(fid=76355, name='Île-de-France', geometry=<MULTIPOLYGON (((2.319 48.788, 2.315 48.789, 2.31 48.785, 2.303 48.785, 2.29...>),

 Region(fid=76356, name='Île-de-France', geometry=<MULTIPOLYGON (((2.32 48.771, 2.312 48.772, 2.308 48.779, 2.31 48.785, 2.315...>),

 Region(fid=76358, name='Île-de-France', geometry=<MULTIPOLYGON (((2.303 48.811, 2.304 48.805, 2.292 48.796, 2.272 48.794, 2.2...>),

 Region(fid=76359, name='Île-de-France', geometry=<MULTIPOLYGON (((2.229 48.774, 2.227 48.776, 2.227 48.782, 2.235 48.786, 2.2...>),

  ...

I will include a method to quickly do things like within 5km etc, maybe something like:

regions.catalog['geometry'].near(paris, 5)