FOSM - volunteer required - map renderer

[80n]

Following the pledge of some financial support, I'd like to get a map rendering server in place for FOSM.

In order to achieve this I'm looking for a volunteer who would be able to install, configure and manage a map rendering service.  The obvious choice would be mapnik but if you have a preference for some other rendering software (TurtleEggs, Tiles@Home, etc) then that may be a possibility instead.

Please get in touch if you can help.

And if you don't know anything about map rendering but would still like to help in some way then please also get in touch, there are many tasks that need to be done over the next three months.

80n

[john whelan]

One approach might be to make city size downloads available then render these on the local machine with Maperitive.  If you could work out a process for compressing the OSM file and decompressing it locally, ie a script so it was reasonably user friendly it might well lower the server render load and internet bandwidth requirements.

Cheerio John

[Mike Dupont]

Hi,
Where will this run? what os? etc. you can add my ssh keys and setup me an acount,
https://launchpad.net/%7Ejamesmikedupont/+sshkeys

right now I am a bit over booked, but interested in helping.

I saw this on colombia, tilebot  looks really interesting
http://osm.ofi.co/status.php

mike

--
James Michael DuPont
Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org

[Sam Vekemans]

groundtruth getdata -m coordinates
(the script is on my wiki user page)

it's command line, windows (dos) ... simple for a non-programmer like
me to understand.
... the user just needs to set the co-ords. :)

Now that i think of it ... it shouldn't be that hard to just make a
full planet script to get the 1x1 tiles :)

... but for specific cities, manually geeting a box is needed...
lambertus did the heavy work of figuring out nice boxes anyway..

cheers,
Sam

--
Twitter: @Acrosscanada
Blogs: http://acrosscanadatrails.posterous.com/
http://Acrosscanadatrails.blogspot.com
Facebook: http://www.facebook.com/sam.vekemans
Skype: samvekemans
IRC: irc://irc.oftc.net #osm-ca Canadian OSM channel (an open chat room)
@Acrosscanadatrails

[Andrew Harvey]

http://wiki.openstreetmap.org/wiki/File:OSM_Components.png

So with reference this this diagram, FOSM is currently set up with the
PostgreSQL Backend supplying API 0.6? I am assuming that planet
dump/diffs will be provided, and then we just need to set up a server
to fetch these dumps/diffs to do what is in the Mapnik box on the
diagram (load into postgis, render tiles using Mapnik with mod_tile
and serve the tiles)? This shouldn't be too hard. I haven't looked
into mod_tile before, so I'll have a look into in.

Also where does the rails port fit in with this, and is it set up on fosm?

[Matija Nalis]

On Sun, Dec 19, 2010 at 01:47:11PM +0000, 80n wrote:
> Following the pledge of some financial support, I'd like to get a map
> rendering server in place for FOSM.
>
> In order to achieve this I'm looking for a volunteer who would be able to
> install, configure and manage a map rendering service. The obvious choice
> would be mapnik but if you have a preference for some other rendering
> software (TurtleEggs, Tiles@Home, etc) then that may be a possibility
> instead.

If that is some GNU/Linux server, I think I could.
(I hope it might be Debian Squeeze or something similar as I'm most familiar
with those; while I should be able to manage other variants too, it might be
quite slower / more time demanding...)

Quick proof of concept (just Croatian map on slow server with not much
bandwidth nor disk space) can be seen at:
http://osm-mapnik.torres.voyager.hr/medium_my.html

The slippymap interface could be more like OSMs of course, this was just
quickest one to implement to show it works...

(note: it uses mod_tile+renderd to render on demand [as the osm.org site
does], so if you get broken image links just wait dozen seconds and
zoom out/zoom in, and they should be there, rendered just for you)

> Please get in touch if you can help.

Contact me directly on e-mail (as I read it more often then lists) if you
haven't found better volunteer yet.

> And if you don't know anything about map rendering but would still like to
> help in some way then please also get in touch, there are many tasks that
> need to be done over the next three months.

I'd like to help in setting up / sysadmining tasks as my free time permits
(which is unfortunately variable and prone to run out when it is most
inconvenient, mind you...)

--
Opinions above are GNU-copylefted.

[Matija Nalis]

So, what is the status of browsable FOSM map?

I repeat my offer to implement it on fosm.org if it is wanted...

[Elizabeth Dodd]

On Sat, 12 Mar 2011 15:13:53 +0100
Matija Nalis <mnalis-opens...@voyager.hr> wrote:

> So, what is the status of browsable FOSM map?
>
> I repeat my offer to implement it on fosm.org if it is wanted...
>

Yes, some news on this would be appreciated

Liz

[80n]

Sorry folks, I've been very busy with other stuff for the past couple of months.

As soon as I get enough time I'll be setting up a Debian based server which can be used as the rendering platform.

Matija, I would like to accept you offer of help and will contact you when I begin to setup the server.

80n

[Elizabeth Dodd]

On Sun, 13 Mar 2011 08:04:11 +0000
80n <80n...@gmail.com> wrote:

> Sorry folks, I've been very busy with other stuff for the past couple
> of months.
>
> As soon as I get enough time I'll be setting up a Debian based server
> which can be used as the rendering platform.
>
> Matija, I would like to accept you offer of help and will contact you
> when I begin to setup the server.
>
> 80n

thanks for the update 80n

[Matija Nalis]

On Sun, Mar 13, 2011 at 08:04:11AM +0000, 80n wrote:
> > Matija Nalis <mnalis-opens...@voyager.hr> wrote:
> > > So, what is the status of browsable FOSM map?
> > > I repeat my offer to implement it on fosm.org if it is wanted...
> > >
>

> Sorry folks, I've been very busy with other stuff for the past couple of
> months.
>
> As soon as I get enough time I'll be setting up a Debian based server which
> can be used as the rendering platform.
>
> Matija, I would like to accept you offer of help and will contact you when I
> begin to setup the server.

Great, whenever you're ready.
Do you perhaps have any estimate when that might be (approximately)?

I ask because it would help me organize time (so that I do not accidentally
dive into some other time consuming tasks or similar exactly at the moment
when I will be needed for FOSM).

[Sam Vekemans]

Help might also be needed on CommonMap, though at this point i'm not
exactly sure where it's needed.
I'll post a note to the osgeo-list once the beta version of the
CommonMap Manual and Feature Catelogue is ready.

Not sure if it's possable to get a hikebikemap rendering system going,
then the code could just be copied.
If 80n needs the 1st mapnik rendering to be created, then i guess that
should be a priority.

I have a feeling that NZ would end-up having their own server API
setup. ... is it easier to setup an independent regional API, then
just have edits syncing back to fosm.org?
... would that lighten the load?

... and perhaps the map could render with a delayed reaction?
Is this possable?

cheers,
sam

--
---
Across Canada Trails - Beyond 2017 - The National Trails Network
Victoria, BC Canada

Twitter: @Acrosscanada
Blog: http://acrosscanadatrails.posterous.com/
Facebook: http://www.facebook.com/sam.vekemans
Skype: 'Sam Vekemans'

Member, CommonMap Inc. http://commonmap.org/
IRC: irc://irc.oftc.net #CommonMap
Also find us on Facebook, Twitter and LinkedIn

[80n]

On Sat, Apr 2, 2011 at 3:30 PM, Matija Nalis <mnalis-opens...@voyager.hr> wrote:

Great, whenever you're ready.
Do you perhaps have any estimate when that might be (approximately)?

I'm still busy but I may have some time soon.  I have to set up a server for another purpose within the next couple of weeks, so if I'm not too stretched I may be able to set up the fosm render server at same time.

If it doesn't happen within the next two weeks then I know I'll be busy again towards the end of April, so it would then be mid May.  That's about the best prediction I can make at the moment.

It is getting important to do something though.  The number of edits on the fosm server is starting to climb and people will want to see their work getting rendered.

 

[80n]

On Sat, Apr 2, 2011 at 4:04 PM, Sam Vekemans <acrosscan...@gmail.com> wrote:

I have a feeling that NZ would end-up having their own server API
setup. ... is it easier to setup an independent regional API, then
just have edits syncing back to fosm.org?
... would that lighten the load?

Sam
 

A distributed osm database would be a really nice thing to have.  It's not easy though.  Just getting the old osm database to sync with fosm threw up all kinds of interesting issues.   Having a regional server could be easier in some ways, especially if it's an island, but for a region that has land borders there are all kinds of issues that would need to be thought through.

 

... and perhaps the map could render with a delayed reaction?
Is this possable?

Delayed rendering is easy ;)  It's getting real-time rendering that is the hard thing.

 

[andrzej zaborowski]

If someone has a spare harddisk I could install it in my server and it
could do as a temporary tile.fosm.org although the link is not super
speedy. I'd then just need your public key and desired login and you
could try setting it up. Right now I'm hesitant to let postgres and
tilecache on the same disk with the rest of my data, but the machine
has relatively good computing power.

Cheers

[John Smith]

On 30 April 2011 10:31, andrzej zaborowski <bal...@gmail.com> wrote:
> If someone has a spare harddisk I could install it in my server and it
> could do as a temporary tile.fosm.org although the link is not super
> speedy. I'd then just need your public key and desired login and you
> could try setting it up. Right now I'm hesitant to let postgres and
> tilecache on the same disk with the rest of my data, but the machine
> has relatively good computing power.

Do you want money for a drive, or actual hardware?

I would have thought it'd be cheaper to buy hardware locally than have
it shipped internationally...

[andrzej zaborowski]

Yeah, probably. But either will work and I can later pass the disk to
someone with a better connection/hardware when it's available. (it
could be a laptop drive or a microdrive / CF card too -- I don't know
what their usual performance is though.. I think CF cards should be
fast?)

Cheers

[John Smith]

On 30 April 2011 16:21, andrzej zaborowski <bal...@gmail.com> wrote:
> Yeah, probably. But either will work and I can later pass the disk to
> someone with a better connection/hardware when it's available. (it
> could be a laptop drive or a microdrive / CF card too -- I don't know
> what their usual performance is though.. I think CF cards should be
> fast?)

CF is relatively slow and low capacity compared to SSD (Solid State
Drives), the downside of SSDs is high failure rate, Seagate put it at
above 30%...

A suitable rack case of 12+ HDDs in the right configuration offers a
lot of IO per second...

[Mike Dupont]

Guys, have you looked at using the hypercube for rendering at all?
or wikipedia toolserver?
mike

--

[Mike Dupont]

I have figured out how to host the tiles on archive.org, it works. now we have unlimited harddrive space there.
i dont understand why you guys dont want to use existing infrastructure that is dedicated to free software/ open source/ open data projects?
mike

[John Smith]

On 1 May 2011 17:09, Mike Dupont <jamesmi...@googlemail.com> wrote:
> I have figured out how to host the tiles on archive.org, it works. now we
> have unlimited harddrive space there.
> i dont understand why you guys dont want to use existing infrastructure that
> is dedicated to free software/ open source/ open data projects?

One problem I foresee if you'll still need some way to keep rendered
tiles up to date, as I'm assuming archive.org only does hosting,
alternatively it might be a good idea to see if they can spare some
CPU cycles to render tiles as well as the data is updated.

[Mike Dupont]

Well, we can all render things, I am working on porting the mapnik rendering rules to standalone without db usage.
For CPU there are many options,
I have many many accounts on servers for free software projects. Toolserver.org comes to mind
see my list here http://h4ck3rm1k3.posterous.com/hello-world-another-host-another-post

Liz suggested that we get tiles at home to upload to archive.org that would also work, but I think we can also just get accounts on toolserver.org or use the ibm linux mainframe host, you can at least write to the different hosting and some might sponsor you. we have gotten sponsorship of one server for flossk from rackspace.

the issue is this, when you use archive.org you need to prepare all the tiles ahead of time, the update interface is not that easy to use, it has not been automated yet. I will try and debug the api and figure it out. But for now it makes sense to render one city at a time and upload all the tiles there.

It should also be possible for use to just render the tiles changes and post them via git, let me show you how.
http://repo.or.cz/w/KosovoCCOpenStreetMap.git/blob/70def061f9e0305756e803f01023f1be6a088fc6:/Municipalities/Klina/Documents/KlinaBuffersImmapIntersect/9/285/323.png

this is one tile, and hosted on the git server, the prefix is
http://repo.or.cz/w/KosovoCCOpenStreetMap.git/blob/70def061f9e0305756e803f01023f1be6a088fc6:/Municipalities/Klina/Documents/KlinaBuffersImmapIntersect

and you can use that as the TMS directly without even uploading the data to archive.org
tms:http://repo.or.cz/w/KosovoCCOpenStreetMap.git/blob/70def061f9e0305756e803f01023f1be6a088fc6:/Municipalities/Klina/Documents/KlinaBuffersImmapIntersect/{zoom}/{x}/{y}.jpg

If you wanted to check in an updated version it would have a new prefix,
Let me do that, I will rerender with different options and higher zoom, it will take a while.
gdal2tiles.py -t "Klina test" --zoom=1-21 --resampling=cubicspline --s_srs EPSG:4326 KlinaBuffersImmapIntersect.jpeg'

I think we dont need the whole world, if everyon renders the places interesting to them, we should be good.

what we need is a dynamic proxy server that will merge tiles from many hosts and sources into one.

that would be a list of what hosts have what tiles in what depth and when they are rendered. each tile would be send to the best source,
that could also be built into open layers and updated on a regular basis, it should be able to be precalculated and also distributed via git.

An alternative would be a local proxy that you could run on your own server. There might be a way to merge may different versions on the same host with git, git stores normally each unique version of the file by a sha1 hash.

Otherwise you could use a CDN like edgecast or something, maybe one would be willing to host?

[andrzej zaborowski]

On 1 May 2011 11:20, Mike Dupont <jamesmi...@googlemail.com> wrote:
> It should also be possible for use to just render the tiles changes and post
> them via git, let me show you how.

This sounds like a quite bad idea. Map tiles are a product of
rendering data, they're like binaries, not source code. Versioning
autogenerated content is usually avoided but in this case it's a
necessity. You'd quickly fill up even the most generous host trying
to keep subsequent versions of tiles for any reasonably sized area. I
also think archive.org is doing a great service to the society and
it'd be a shame to waste their resources for keeping autogenerated
content, especially autogenerated and quickly becoming outdated
content (an OSM tileserver can generate gigabytes of new data every
minute), you know it kills hard drives quickly. Also there's a reason
modtile renders on demand.

Cheers

[John Smith]

Some times I wished in the past to see previous version of tiles to
see what has changed, yes it'd be a massive waste of space, but it'd
still be nice.

Perhaps instead of focusing on rendering server side we should be
focusing on rendering client side.

[Mike Dupont]

On Tue, May 3, 2011 at 2:30 AM, andrzej zaborowski <bal...@gmail.com> wrote:
> On 1 May 2011 11:20, Mike  Dupont <jamesmi...@googlemail.com> wrote:
>> It should also be possible for use to just render the tiles changes and post
>> them via git, let me show you how.
>
> This sounds like a quite bad idea.  Map tiles are a product of
> rendering data, they're like binaries, not source code.  Versioning
> autogenerated content is usually avoided but in this case it's a
> necessity.  You'd quickly fill up even the most generous host trying
> to keep subsequent versions of tiles for any reasonably sized area.

you can purge old versions in git as well, even on the server
http://www.kernel.org/pub/software/scm/git/docs/git-filter-branch.html

see also
http://stackoverflow.com/questions/930612/git-remove-oldest-revisions-of-a-file

and with submodules you can reference other repositories and select
versions, that can be used to weave many repos together.

> also think archive.org is doing a great service to the society and
> it'd be a shame to waste their resources for keeping autogenerated
> content, especially autogenerated and quickly becoming outdated
> content (an OSM tileserver can generate gigabytes of new data every
> minute), you know it kills hard drives quickly.  Also there's a reason
> modtile renders on demand.

WIth archive org we can replace files and keep only the new ones.

>
> Cheers

[Mike Dupont]

as I said we can purge old versions, and that is in git. If we use
archive org we can replace files directly.

> Perhaps instead of focusing on rendering server side we should be
> focusing on rendering client side.

yes, I think we should work on client side rendering with many many
people helping , tiles at home etc.
I think we could even render only the changed layers when updating the
repository.
we can also run mapnik at home, I am experimenting with that.