Is there a way to read from an OSM import kept in the PostgreSQL instead of a `*.pbf`?

[Sergii]

Hello everyone!

I wonder if there is a possibility in the `imposm3` to read from an OSM import kept in the PostgreSQL instead of a `*.pbf`?

We use `openmaptiles` which is built using the `imposm3`. We want to change the dataflow to render map tiles starting not from the `*.pbf`, but from an already running OSM extract in the PostgreSQL. I have described our case here:

https://github.com/openmaptiles/openmaptiles/issues/486#issuecomment-498242786

Thanks for any ideas! :)

[Imre Samu]

my 2c:

> I wonder if there is a possibility in the `imposm3` to read from an OSM import kept in the PostgreSQL instead of a `*.pbf`?

As I know - only OSM input possible.

>We use `openmaptiles` which is built using the `imposm3`. We want to change the dataflow to render map tiles starting not from the `*.pbf`, but from an already running OSM extract in the PostgreSQL.

in theory - you can create a simple PostgreSQL view on osm2pgsql tables - and re-implementing  imposm3 tables, but  this way is the hacker way ;   lot of hacking, debugging, performance tuning ...  

Best,

Imre

--
--
_______________________________________________
Imposm mailing list
imp...@googlegroups.com
http://groups.google.com/group/imposm

---
You received this message because you are subscribed to the Google Groups "imposm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to imposm+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/imposm/276d3ddb-a56f-43d7-880a-39195f0e6140%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sergii Golubiev]

Hi, Imre

Thanks for your reply!

>in theory - you can create a simple PostgreSQL view on osm2pgsql tables - and re-implementing  imposm3 tables, but  this way is the hacker way ;   lot of hacking, debugging, performance tuning ...  

Did you mean to imitate (port/re-implement) the schema that is being produced by `imposm3` with PostgreSQL views? If so, I do agree with you that it would be a lot of hacking. I would need to port the behavior of e.g. `area` and `validated_geometry` fields, generalized tables, etc. IMHO, it would be nontrivial in implementation and then making it produce results close to the imposm's ones. And later - it would require further support.

It seems to me that changing the input of the imposm3 from `*.pbf` to OSM can be an easier task to implement and support, as it does not touch mapping internals.

For this case, it would be ideal if there was such option "out of the box" :)

пн, 3 июн. 2019 г. в 17:34, Imre Samu <pella...@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/imposm/CAJnEWwkjeFngnm9s2H4vnkD-%3Dgub4k6QHVqM2Fbwg6atBwK4pg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

С уважением,

Голубев Сергей,

веб-разработчик ЛУН.ua

[Jeff McKenna]

On 2019-06-03 10:34 AM, Imre Samu wrote:
>
> in theory - you can create a simple PostgreSQL view on osm2pgsql tables
> - and re-implementing  imposm3 tables, but  this way is the hacker way
> ;   lot of hacking, debugging, performance tuning ...
>

I implemented such a thing for the MapServer community, which you can
find at:
https://github.com/mapserver/basemaps/blob/master/contrib/osm2pgsql-to-imposm-schema.sql
which is a part of the process at:
https://github.com/mapserver/mapserver/wiki/RenderingOsmDataWindows

-jeff


--
Jeff McKenna
MapServer Consulting and Training Services
https://gatewaygeomatics.com/

[Imre Samu]

Hi Sergii,

> It seems to me that changing the input of the imposm3 from `*.pbf` to OSM can be an easier task to implement and support, as it does not touch mapping internals.

>For this case, it would be ideal if there was such option "out of the box" :)

If you are using  standard "local" OSM server (PG) with "local" API  then

- you can create your own .pbf dump ; and/or you can create osm changesets (  Minutely, Hourly, or Daily diffs )  -  and Imposm3 can read this files.

imho - this is the simplest method.

This is How the current OSM  ecosystem is working. 

But any other method is ok,  

If you can create an osm changesets  - the imposm3 can read it.

"Imposm can keep the OSM data up-to-date by importing changes from OSM changes files. It needs to cache a few more information to be able to update the database from diff files."

- https://imposm.org/docs/imposm3/latest/tutorial.html#updating

- https://wiki.openstreetmap.org/wiki/OsmChange
 

Best,

 Imre

To view this discussion on the web visit https://groups.google.com/d/msgid/imposm/CANo%3DrzGZmSz_qHurCwOAwr5ug55c8cTXQzDGF1RLrNVEXm690g%40mail.gmail.com.

[Imre Samu]

Hi Jeff

>I implemented such a thing for the MapServer community, which you can ...

thanks for the info!  

Best,

  Imre

--
--
_______________________________________________
Imposm mailing list
imp...@googlegroups.com
http://groups.google.com/group/imposm

---
You received this message because you are subscribed to the Google Groups "imposm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to imposm+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/imposm/ca5c514e-2f06-2f39-ae6d-67fad2c55c12%40gmail.com.

[Sergii Golubiev]

>I implemented such a thing for the MapServer community

Hi, Jeff! Thanks! I will look into it!

вт, 4 июн. 2019 г. в 14:41, Jeff McKenna <jeffm...@gmail.com>:

--
--
_______________________________________________
Imposm mailing list
imp...@googlegroups.com
http://groups.google.com/group/imposm

---
You received this message because you are subscribed to the Google Groups "imposm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to imposm+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/imposm/ca5c514e-2f06-2f39-ae6d-67fad2c55c12%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Sergii Golubiev]

Hi, Imre!

Thanks for your ideas!

>you can create your own .pbf dump

We had this idea too: to share OSM data via the `*.pbf` dump. It is not an optimal way, but straightforward and working.

But I don't know whether I can create an appropriate dump with the OSM planet dumping tool (https://github.com/zerebubuth/planet-dump-ng). In our case, we are running not the whole OSM DB replica (with users, changesets and so on). But we have DB with the OSM data in the `pgsnapshot` schema:

https://wiki.openstreetmap.org/wiki/Databases_and_data_access_APIs#pgsnapshot

>The pgsnapshot schema is a modified and simplified version of the main OSM DB schema which provides a number of useful features, including generating geometries and storing tags in a single hstore column for easier use and indexing.

I am currently researching in this way.

вт, 4 июн. 2019 г. в 15:17, Imre Samu <pella...@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/imposm/CAJnEWwmKM75fk693TBKZQ75A8o2iEMfHzo_9NCb%3DwWX3FNHG5Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Sergii Golubiev]

Finally, I succeeded in creating a `*.pbf` dump from our running OSM DB in the `pgsnapshot` schema.

I totally failed with the https://github.com/zerebubuth/planet-dump-ng tool, because it seems to be designed only for the full OSM DB (with users, changesets, etc.) - not the case we had.

Accidentally I found that the tool we already use that populates our DB from a Geofabrik's `*.pbf` - https://github.com/openstreetmap/osmosis - could also do the reverse operation. This article - http://marcusjenkins.com/plumbing-with-openstreetmap-osmosis/ - was very helpful, because it was not so trivial to do the reverse operation.

It was very fun when after a couple of days of research I found that the tool we already use is the right one to be used :)

Imre and Jeff, thanks for your sharing your ideas and experience!

We will try to implement a prototype where we will chain a `pgsnaphot` OSM PostgreSQL instance with the `imposm3` via a `*.pbf` extract dumped with the osmosis tool: pgsnapshot -> *.pbf -> imposm3

Best regards,

Sergii

ср, 5 июн. 2019 г. в 10:57, Sergii Golubiev <g...@lun.ua>: