[postgis-users] PSC Vote: Keep or drop Flatgeobuf in PostGIS 3.2.0

[Regina Obe]

This is a PSC vote, but we would like some feedback on this from packagers
and users as such comments will sway our vote.

We have two blockers that center around the new FlatGeoBuf format.

https://trac.osgeo.org/postgis/ticket/5005 (this one is easily
replicatable)

https://trac.osgeo.org/postgis/ticket/5014 (this one I can only replicate
with the cowbuilder setup Bas Cowenberg provided)

both I think are manifestations of the same problem how the header is
derived and what it's doing with numeric and geometry fields.

I've taken a stab at troubleshooting and fixing, but did not have much luck.
That said, if anyone is willing to help fix that would be great and fix
within a 1 to 2 week time period.

If not I feel that we really need to take it out of our PostGIS 3.2.0
release (which will be going on to 3.2.0beta2).

I'd like to release PostGIS 3.2.0beta2 in about a week or so with flatgeobuf
fixed or removed. If removed, we'll push flatgeobuf to PostGIS 3.3.0
cycle.

Thanks,
Regina


_______________________________________________
postgis-users mailing list
postgi...@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/postgis-users

[Darafei "Komяpa" Praliaskouski]

Hi,

I have not seen flatgeobuf in the wild, and I believe it can be safely removed. 

The current implementation is impaired by Postgres' life choices of 1GB limit and thus not usable for any data, just size-limited subset. ogr2ogr seems like a better suited place for it to reside.

I'm -0 on adding flatgeobuf to core, and -1 on releasing with known crashers. This would converge to "remove if nobody can fix crashers".

postgis-devel mailing list
postgi...@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/postgis-devel

[Regina Obe]

FWIW it’s already in GDAL since 3.1 and yah GDAL is a better home since it doesn’t have the  1GB PostgreSQL limitation

 

https://gdal.org/drivers/vector/flatgeobuf.html

 

Also here are OpenLayers and Leaflet examples for those not familiar with the format

 

OpenLayers: https://flatgeobuf.org/examples/openlayers/

 

 

Leaflet: https://flatgeobuf.org/examples/leaflet/

 

Thanks,

Regina

[Bruce Rindahl]

FWIW I say remove it and seriously think about not including it at all.  Looks like you can use the format right now via ogr_fdw using GDAL. 

[Björn Harrtell]

Even though I spent quite a bit of effort on implementing this stuff and I'm sure I can fix the crashers I agree with the arguments to remove it. That is, 1GB limit is really bad and better to use GDAL which has a well maintained impl of it.

If there was a way to stream in and out binary with custom encoding and no size limit (i.e COPY with custom/ext binary format) it could make sense but I don't think that is going to happen any time soon.

Oh well, it was fun. Some of it. 😂

PS. ST_AsGeobuf should be deprecated/removed too - it's even less useful IMHO.

PS2. I do still believe in FlatGeobuf and it is used in production. ;)

/Björn

[Regina Obe]

As mentioned on IRC just reiterated here. 

 

I would like to see ST_AsFlatGeoBuf in there. Here are my reasons.

 

1) To Bruce’s point that sure ogr_fdw can read it, I’m more interested in the writing of it (which at least I can’t do from scratch with ogr_fdw) and getting out of the database onto the command line is not one of my great ambitions in life.

 

2) The fact it is a lighter weight format than ST_AsGeoJSON which has the same 1GB limitation.

Or as mentioned https://flatgeobuf.org/ a better shapefile format (for < 1GB)

And the fact that FlatGeobuf is about 50% lighter than GeoJSON

From site:

  (shapefile 1, FlatGeoBuf 0.77,  GeoJSON  1.2)

 

Means I can stuff much more data in the 1GB limit than I can in GeoJSON.

Though not sure about how that places with the other attribute data if they are compacted as efficiently.

 

3) Given it is  demonstrated it can work fine with leaflet and openlayers means it is possible to hook it into something like

https://github.com/CrunchyData/pg_featureserv

 

as an alternative format to geojson (not saying you should Martin Davis – just throwing it out there J )

 

4) One more reason to fight that 1GB limit – as postgis raster is impacted by that too.

 

 

Thanks,

Regina

[Bruce Rindahl]

Sufficiently chastised.  Based on Regina's comments, I might have a use case for it.

[Martin Davis]

On Wed, Nov 24, 2021 at 3:10 PM Regina Obe <l...@pcorp.us> wrote:

3) Given it is  demonstrated it can work fine with leaflet and openlayers means it is possible to hook it into something like

https://github.com/CrunchyData/pg_featureserv

as an alternative format to geojson (not saying you should Martin Davis – just throwing it out there J )

Agreed, and already thinking about doing this.  If it's built into PostGIS this should be easy to do :).   It could be done even if it's not in PostGIS, if there is a Go lib to do it (?).  But I prefer using PostGIS-native formats.

Nice that it's in Leaflet and OpenLayers already.  That makes a pretty solid case for supporting it.   And I'd be very interested in seeing what the performance characteristics are.

Consider this a +1 for keeping it in.

[Björn Harrtell]

Thanks for the nice motivation.

Issue #5005 is no more. #5014 is strange and I'm stuck in unable to reproduce land.

[Björn Harrtell]

Just a FWIW,

FlatGeobuf is actually not prioritizing space optimization, instead designed for simplicity and speed. Even so, it can be smaller than both shapefile and geojson but it depends on data. But so, it can make sense to compress if space optimization is required.

The primary reason for why FlatGeobuf is fast is the use of Flatbuffers which is not far from C structs (i.e direct access, no decode, no extra allocations needed) but has a stable platform independent wire format. If created without index FlatGeobuf can be both written and read as stream, which makes it very fast as a system to system transfer format. If written with an index a FlatGeobuf can be accessed with a bbox filter very efficiently because it has a guaranteed balanced Hilbert R-tree, so can be traversed with forward seek and found data locality is good because data is ordered by the spatial index. Unfortunately bytea input/output does not allow full advantage of these properties - something like COPY would be better (since it can be streamed).

/Björn