Infer Parquet GEOGRAPHY algorithm from GeoParquet

[Sylvain Lesage]

The new GEOGRAPHY Parquet logical type has an intriguing field called 'algorithm' [1]. I'm wondering if there is an equivalent in GeoParquet, ie if we can infer this value while converting from Geoparquet to a native GEOGRAPHY Parquet column?

[1] https://github.com/apache/parquet-format/blob/master/Geospatial.md#edge-interpolation-algorithm

[Sylvain Lesage]

OK, I understood where the 5 values for `algorithm` come from, they are defined in GeoArrow: https://geoarrow.org/extension-types.html#extension-metadata, as possible values for the `edges` field.

But the Geoparquet specification clearly does not allow these values, and only supports `planar` or `spherical` for `edges`.

So: the answer is: we cannot infer the algorithm, and we must let the field empty in the GEOGRAPHY logical type.

[Chris Holmes]

Sorry for the slow response. My exact memory of this is a bit rusty, but hopefully someone will correct where it's wrong. 

So in Geoparquet we just defined Spherical and and Planar, since practically that's all the information you really need to know. We never had any users ask for anything more, and none of the implementations of spherical provided any documented info on the algorithms they supported. When the Iceberg PMC was looking at it they had some people who wanted to go really deep with this, and in my opinion they ended up over-specifying this. By which I mean that the philosophy of GeoParquet community has been to encapsulate what already exists, not to try to predict new potential behaviors (though to build hooks for potential future extensibility, like we did with the geometry type). There are not actual implementations of the algorithms, so it's not something that we'd specify. 

But we're not the iceberg spec authors, and in our effort to influence the spec from the side we decided that this one wasn't important enough - it's a bit over-specified, but it doesn't _hurt_ anyone. It's just some of those may never be used. 

The iceberg spec is the center of it, and then it's flowed to GeoArrow, and Parquet.

We'll aim to clarify this in a next version of the spec, and likely will just add the fields.

I'm not expert enough to recommend what to do in conversion, hopefully others will sound in. I suspect you could just use 'spherical' both ways and be safe, since all the implementations that we know of are spherical, and the differences between the algorithms are afaik not super major unless you need super high precision. 

C

To unsubscribe from this group and stop receiving emails from it, send an email to geoparquet-commu...@cloudnativegeo.org.