Map server testing

[Tim Sheerman-Chase]

I have most of a map server implemented in python/C++. It's very much in
testing at the moment, so edits won't be retained in the long term.

Register an account: https://tim.noip.me:8012/

API URL: https://tim.noip.me:8012/api

Is there any interest in hosting/using this as an alternative to what we
have?

Tim

[o...@k3v.eu]

Hi Tim,

I created an account. I will try some editing tomorrow evening.

I still think there is a need for an alternative to OSM so I would like
the project to keep going. I assume the DB size is what would stop us
using much cheaper hosting as the traffic can't really be that much.

Kevin

[Tim Sheerman-Chase]

Kevin,

Thx for comments. I'm not comfortable contributing to the fosm server if
80n isn't going to have time to maintain it or update it. I think
another platform would be good.

The current planet data with only a fairly small history is 500Gb in a
postgis database. The GT.M databse on fosm.org is about 700Gb but this
has a more complete history of changes. I agree the bandwidth would not
normally be very much. Mostly importing and exporting dumps, etc. The
storage is basically pushing us towards co-location or unmanaged
dedicated as most hosts don't provide this much storage for a reasonable
price.

I would host a server on my home connection but I'd like something a
little more reliable! The unmanaged dedicated services with 1TB HDD from
delimiter.com for $49/mo
https://www.delimiter.com/atlanta-dedicated-servers/ might work or a
custom solution for $39/mo from serverpronto.com but it's still
expensive. Co-location looks to be about the same price(?) plus setup
fees plus getting hardware. Can anyone pull strings and find a cheaper
option?

Tim

[Ross Scanlon]

Hi All,

I've taking over payment for the api server for the next twelve months.

By the time it comes around again I hope to have transferred it to a
server in my network.

This will be the same network as the map server is currently on.

I have issues with the gtm script to load the planet data into the
database, this is the only thing that stopped me transferring it before
this. The issue is that the script gives an error with object already
in database, this is from memory as the server is not online at the
moment and I'm away.

Traffic is very low and yes we could do with more database storage. The
server I would be using has 2gb for the data alone as opposed to the
current 1gb total.

Tim. Do you have the code on git?

I'd rather use python/C++ as I'm much more familiar with these than gtm.

Cheers

Ross

[Arthur Ingram]

There is a c++, python, and nodejs interface to GTM MUMPS

https://github.com/dlwicksell/nodem NodeJS

https://www.osehra.org/blog/gtm-binding-c++ for c++

ruby, php, java

Also their is a fork http://yottadb.zohosites.com/home.html built on top of GTM

art

[Arthur Ingram]

There is also QEWD javascript frame work http://qewdjs.com

On Sunday, October 15, 2017 at 12:42:47 PM UTC-7, TimSC (Another Old Account) wrote:

[o...@k3v.eu]

On Mon, 16 Oct 2017 00:12:07 +0100
Tim Sheerman-Chase <map...@sheerman-chase.org.uk> wrote:

> Kevin,
>
> Thx for comments. I'm not comfortable contributing to the fosm server
> if 80n isn't going to have time to maintain it or update it. I think
> another platform would be good.
>
> The current planet data with only a fairly small history is 500Gb in
> a postgis database. The GT.M databse on fosm.org is about 700Gb but
> this has a more complete history of changes. I agree the bandwidth
> would not normally be very much. Mostly importing and exporting
> dumps, etc. The storage is basically pushing us towards co-location
> or unmanaged dedicated as most hosts don't provide this much storage
> for a reasonable price.
>
> I would host a server on my home connection but I'd like something a
> little more reliable! The unmanaged dedicated services with 1TB HDD
> from delimiter.com for $49/mo
> https://www.delimiter.com/atlanta-dedicated-servers/ might work or a
> custom solution for $39/mo from serverpronto.com but it's still
> expensive. Co-location looks to be about the same price(?) plus setup
> fees plus getting hardware. Can anyone pull strings and find a
> cheaper option?
>

@tim Having the API written in something more modern and maintainable
is a no-brainer. Did a short test this evening, tried a few amendments,
a decent sized delete and add and it all seemed to work okay and upload
and download speeds were good.

@ross Sounds great. Do you have a Paypal account or something if people
want to contribute.

Kevin

[Ross Scanlon]

Hi All,

If you wish to make a donation for the server cost this year.

payment @ freestreetmap . com


I am happy to pay for it this year and I'll look into putting a new
server or upgraded dedicated server next year.

The delimiter servers seem a good price for what you get.


Cheers

Ross

[Tim Sheerman-Chase]

Ross,

nice move about renewing for now. It would take quite a bit to move
operations to a new (but tested) platform.

The software is just beginning to stabilize. I need to write more
documentation about installation. Currently there are three main repos:

Import tool (python): https://github.com/TimSC/osm2pgcopy

Low level database code (C++/swig): https://github.com/TimSC/pgmap-query

Website and API (python/django): https://github.com/TimSC/pycrocosm

Where do you want to start? (I'll document that part first.) If anyone
wants to contribute, send me pull requests. There is a fair bit of work
left to do on the API (and website), including making everything easier
to install. Perhaps docker would simplify things? Not sure, not much
knowledge of that yet... I also want to merge the osm2pgcopy
functionality into pgmap-query because its really slow... Oh, btw, these
repositories pull in other submodules so check them out with git
--recursive.

1TB storage would be (about) the minimum to get anything done, but it
would not have much space for backups or maintenance. Agree that 2TB
would be more practical. Ross, is hosting my implementation on your
network an option? or you would be leaning towards 3rd party hosting?

Arthur, I was aware of the C interface to GT.M. If we were to have
updated the fosm server in place, it would have been elegant to update
the website while keeping the database the same (for a time anyway).
However, I'd like to update both the website and database to something
other than GT.M eventually. I've already implemented a
python/C++/postgis solution, so now we might as well move on.

I have not implemented the XAPI functionality at all. I have some ideas,
perhaps using partial indices in postgis.

Tim

[Ross Scanlon]

Hi Tim,

Where to start?

I don't have access to the spare server at the moment and probably won't
until mid November. It's not on line, nor powered up and I'm not where
it is. I'd also suggest that it would be mid December before I could
look at putting it on line.

Documentation, I'd look at them in the order you have below. This is
going to be basically how we would need to use them.

As for installation I'd be quite happy with just a manual install of
each item as you already show on git. Maybe later look at creating an
auto installation process.

I'd look at hosting your implementation as I can see that being easier
to use/maintain than the current gt.m implementation. I have installed
the gt.m implementation on the server at the moment but as I said there
are problems getting it up and running. However we need to look at dev
on a separate server (maybe where you have it now) as well as the main
api server.


However a couple of quick questions.

Does it do minutely/hourly/daily diffs? Main reason is I'd still run
the map and nominatim servers separately so need minutely diffs for them.

Can it pull in diffs from the current api server so we can do transfers
up to changeover?

Can we do an easy planet dump?


I may look at one of the 3rd party dedicated servers later on. The
delimiter servers are a better price than what bytemark is but I'll
leave it where it is for now.


Cheers

Ross

[Tim Sheerman-Chase]

Ross,

I just added diffs at https://tim.noip.me:8012/replication/ Currently
they are generated by querying the database but there is no reason why
they can't be cached. In fact this is a good way to keep an extra copy
of the data. (At least I think I've implemented it right, it was not a
feature of my old PHP map server).

Yes, diffs can be applied to the database. So far it is done by a
command line rather than automatically.

Planet dump is supported (but it still takes about 2 days to run). It
looks like this can run in the background since EXCLUSIVE lock for
editing does not require the dump to stop.

I don't think there is a big rush on the server, as long as we have a plan.

The database import tool is somewhat documented:
https://github.com/TimSC/osm2pgcopy The basic process is setting up
postgis, then converting an extract (or planet) to compressed csv files,
then using a python tool do the import. I'm still working on the schema,
so it probably makes sense to start with a small-ish extract since you
would probably need to rerun the import process after any schema
changes. Converting to csv files takes about 2 days for a planet dump,
which is dreadfully slow (python is not good here). Creating indices on
the tables takes 1-2 days for a planet dump. Let me know if you get
stuck. The hardest bit is getting permissions right within postgis...

Tim

[John Smith]

On 18 October 2017 at 16:53, Tim Sheerman-Chase <map...@sheerman-chase.org.uk> wrote:

Planet dump is supported (but it still takes about 2 days to run). It looks like this can run in the background since EXCLUSIVE lock for editing does not require the dump to stop.

most of the planet file would be static, wouldn't it be smarter to export an aussie/UK extracts instead?

[Ross Scanlon]

Hi Tim,

Thanks.

I'll look in to this once I've got the server online.

Looking at whats on git at the moment I don't see an issue with setting it up, it's all straight forward.  If anything crops up I'll let you know.

If you can continue developing what you have at the moment would be great.  Only thing I've seen that I'd look at changing to make configurable is hard coded directories.  eg in pycrocosm/changeset/management/commands/applydiffs.py line 66

Also maybe a comment for each function as to what it's supposed to do.  Sorry I'm an old school programmer and was beaten into me to have comment for every function.  Mind you I don't always do it and usually regret it later.

My intention is to get a new api server up and running but initially just importing from the current api server.  I'll use a similar script to what I use on the map server to apply the minutely diffs.

Once it's stable I'll look at changing over editing similar to what was done when it was forked from osm.  So new id's for new changesets/nodes/ways/relations/diffs probably beginning with 2xx.......  Anything that is edited on the current api server will continue to be updated but id's will not conflict.    Target July 2018.

Then by October 2018 we can turn off the current api server.

I have not seen anything from 80n on this so not sure on his thoughts on possibly changing over.

Also need to look at possibly transferring the domain name.  Assuming we keep fosm.org which I suggest we do.

fosm.org could then be on the the nameservers I use.

May also look at a redesign of the main website.  Possibly an extension to map.fosm.org

Cheers

Ross

[Tim Sheerman-Chase]

We probably should do extracts, perhaps monthly for areas we are particularly mapping. An annual planet dump would be sufficient for backup purposes.

I often wondered if we could somehow make extract diffs...

Ross, well spotted on the hard coded directories. It is like of unavoidable in the website configuration. Django commands (including applydiffs.py) apparently take command line arguments but I've not had time do figure it out.

Comments would be good. An architecture diagram would probably help as well!

"Anything that is edited on the current api server will continue to be updated but id's will not conflict." Diffs are not applied with the same level of validation as API edits (at least at the moment) for speed reasons. Nodes on one server can be deleted that are used within ways on the second server. A quick transition would be safer (or we can add more validation to diffs, I guess). I still suspect we have piles of broken ways in the database from last time...

Keeping fosm.org would be good if 80n is willing. A web page redesign would be good. I could adapt a template or someone with graphic design skills could try a new design...

I suggest we move as soon as we feel confident because we can add new website features and tools.

Tim

[Ross Scanlon]

Hi Tim,

I have been experimenting with pycrocosm just on my laptop at the moment.

When you have a chance could you expand on the installation steps in the README file.

I think I'm understanding it but just looking at where you are coming from/heading with it.

Cheers

Ross

--
You received this message because you are subscribed to the Google Groups "OSM Fork" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osm-fork+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Sheerman-Chase]

I've added some high level comments to https://github.com/TimSC/pgmap

Much more commenting needed of course!

Tim

[Ross Scanlon]

Hi Tim,

I have not looked into pgmap much other than to see how it installs as
the laptop is a little limited in space.

My main interest is in getting https://github.com/TimSC/pycrocosm up and
running.

I can see to install django and obviously need to install nginx and
php5-fpm but I'm wondering where your installation goes from there.

Will not have much time in the next three weeks to look at it.

Cheers

Ross

[Tim Sheerman-Chase]

Ok, I'll update those docs soon. You can actually skip nginx and uwsgi
if you are just doing development or testing because Django has a fairly
decent webserver built it. (python manage.py runserver)

Tim

On 27/10/17 00:10, Ross Scanlon wrote:
> Hi Tim,
>