Server problem

[Ross Scanlon]

Hi Folks,

Apologies for any inconvenience.

It looks like I've had a router failure.

Unfortunately I'm not available to check it and I can not get anyone
else there before Saturday this week.

I'm away until Tuesday or Wednesday next week and that will be the
earliest I can ensure that the router is working.

I can ping the network node but am unable to access the router remotely
so it looks like it has either had a power surge and not reset or failed.


I am also advising that there will be interruptions to the api server on
Friday 20th September 2019 AEST.  They should be only 30 minutes to an
hour and will be random throughout the day.

This is power work to convert the power supply to grid connected solar
and eventually a standalone solar setup with grid backup.

Cheers

Ross

[Tim Sheerman-Chase]

Looks like we are back online :)

Thanks Ross

Tim

[Ross Scanlon]

Yes we are.

I may have to take it offline for a short time either Thursday 3rd or
Friday 4th October (AEST), should only be half hour.

I will advise if I need to.

Cheers

Ross

[Tim Sheerman-Chase]

The server has been offline for a few days?

Tim

[Ross Scanlon]

Hi Tim,

I've only just got back to some where with internet access.

Don't know what's happening with the server as I can't access anything
there (server, router or watchdog) and I'm about 1600kms away.  I'll try
to get someone to look at it over the weekend.

@80n

In the mean time can you please change the ip for fosm.org to:

59.167.220.189

so at least we can provide updates.

Cheers

Ross

[Ross Scanlon]

Server should be back online now.

@80n

No need to change the ip for fosm.org now.

Cheers

Ross

[Tim Sheerman-Chase]

Many thanks!

Tim

[Tim Sheerman-Chase]

Hi all,

I've added basic xapi functionality to the git version of the server. It
does require the bounding boxes of ways and relations to be calculated
which is very time consuming on a conventional HDD. I'm assuming the
live server doesn't have SSD storage? Now might be a time to consider
changing over to that faster storage. I need to see how much more hard
drive space is required. For my xapi API, I currently only support bbox,
one key and a single value or * wildcard. Upgrading the server will
require the "active tables" to be re-imported i.e. everything since the
original "static" global import but that should be a fairly quick
process if we have the relevant diffs.

There is also the possibility that way and relation bboxes can
accelerate standard map queries. However, these would return ways that
overlap with the query region, even if they have no nodes within the
query. (Standard map queries only return ways if there are nodes in the
relevant area.) This can be seen as an advantage or a disadvantage but
it probably won't break anything. Currently this mode is optional in
pycrocosm.

Ross, are we on python2 on the server? Probably worth considering
upgrading to python3 mainly because django is now only supporting that
going forward.

I've added some functionality to track the most active editors as well
as the rate of map queries. I'm not sure if we want to display that to
users but it might be good to see how many editors are active.

If people have ideas for new features, let me know.

I still need to write up some upgrade instructions.

Tim


On 10/08/2020 23:31, Ross Scanlon wrote:
>
> OK.   I should have time after friday as don't have anything planned then.
>
> Cheers
>
> Ross
>
>
> On 11/8/20 1:50 am, Tim Sheerman-Chase wrote:
>>
>>
>> Great. I have knocked around a few ideas in code but I'm quite far
>> from having the code ready for deployment.
>>

[Ross Scanlon]

Hi Tim,

Happy new year to all.

The live server has HDD for the database as 2TB SSD's were not available
for a reasonable price at the time of build.  I have been keeping an eye
out for reasonably priced 2TB SSD's (< $AUD250) two are needed as it's
in a raid array.


Python2 and Python3 are installed on the server. However it's using
Python2 as the default.

Will django 1.11.8 work with python3?


There is currently a cron job on the server which calculates and adds
the bounding boxes for changesets.  This is done twice a day.  It may be
possible to incorporate the way bounding boxes as part of this, as it
checks the max/min lat lon for each node and way in a changeset that is
created or modified and uses these to calculate the bounding box for the
changeset.  It should be possible to expand this to do the same for
relations.

There is slightly less than 1TB available for the database on the server
with 2TB HDD's currently in the raid array.


Cheers

Ross

[Tim Sheerman-Chase]

Yes, SSDs have come down so much in price over the years. Do we need a
crowd funder to make things fair for you Ross to cover expenses?

I was assuming we would be updating django and all dependencies now we
have the opportunity. Everything seems to be fine on the latest django
(3.1.4) when run locally. I'm currently targeting
https://github.com/TimSC/pycrocosm/blob/master/requirements.txt Might
there be a problem upgrading? If we get the live server up to recent
versions, I might deprecate python 2 support since it has reached end of
life.

Why are you tracking changeset bounding boxes? I assume it's for mapnik
rendering? I have a feeling pycrocosm doesn't do this properly and I'm
not surprised you have script. Now would be a good time for me to finish
this functionality.

My new way and relation bbox tracking is available via the web API.

I've been thinking how to do diffs for the purposes of updating
extracts. Standard changeset diffs are only really good for updating a
planet database. This might be useful for checking which map tiles need
regenerating as well. I've not decided on a final approach yet...

I've not actually managed to get a planet to completely update since I
personally don't have a SSD that big but I'm thinking of moving over...
If possible it would be good to get a beta version up and running on my
hardware or Ross's to make sure we are confident in its reliability.

Tim

[Ross Scanlon]

On 7/1/21 1:04 am, Tim Sheerman-Chase wrote:
>
> Yes, SSDs have come down so much in price over the years. Do we need a
> crowd funder to make things fair for you Ross to cover expenses?

Even if we could get the cost of one covered would be good.   The couple
of companies I normally source from do have regular sales but tend to
limit it to one per customer at the sale price and I want to ensure both
SSD's are the same.

>
> I was assuming we would be updating django and all dependencies now we
> have the opportunity. Everything seems to be fine on the latest django
> (3.1.4) when run locally. I'm currently targeting
> https://github.com/TimSC/pycrocosm/blob/master/requirements.txt Might
> there be a problem upgrading? If we get the live server up to recent
> versions, I might deprecate python 2 support since it has reached end
> of life.

I don't envisage any problem updating.  A quick check last night
revealed no issues with using python3 in place of python2

>
> Why are you tracking changeset bounding boxes? I assume it's for
> mapnik rendering? I have a feeling pycrocosm doesn't do this properly
> and I'm not surprised you have script. Now would be a good time for me
> to finish this functionality.
>

The changeset bounding boxes are for this:

http://fosm.org/changeset/2000012396

> My new way and relation bbox tracking is available via the web API.
> I've been thinking how to do diffs for the purposes of updating
> extracts. Standard changeset diffs are only really good for updating a
> planet database. This might be useful for checking which map tiles
> need regenerating as well. I've not decided on a final approach yet...

Map tiles are marked as dirty when the rendering database is updated
with changeset diffs.


Cheers

Ross

[Tim Sheerman-Chase]

On 07/01/2021 03:41, Ross Scanlon wrote:
> On 7/1/21 1:04 am, Tim Sheerman-Chase wrote:
>>
>> Yes, SSDs have come down so much in price over the years. Do we need
>> a crowd funder to make things fair for you Ross to cover expenses?
>
> Even if we could get the cost of one covered would be good.   The
> couple of companies I normally source from do have regular sales but
> tend to limit it to one per customer at the sale price and I want to
> ensure both SSD's are the same.
>

Ross do you want to set up a crowdfunder to target AUS $300? I'd be
willing to chip in.

>
>>
>> Why are you tracking changeset bounding boxes? I assume it's for
>> mapnik rendering? I have a feeling pycrocosm doesn't do this properly
>> and I'm not surprised you have script. Now would be a good time for
>> me to finish this functionality.
>>
> The changeset bounding boxes are for this:
>
> http://fosm.org/changeset/2000012396

I implemented the changeset bbox algorithm from here
https://wiki.openstreetmap.org/wiki/API_v0.6#Bounding_box_computation
The approach doesn't make much sense to me but it should more or less work?

I'm hoping tracking bboxes and changeset bounds won't have much of a
performance impact...

Tim