OSM - Tile usage policy
!i!
first let me say, that I realy like OSMand, as it is a quite easy way
to access our material but:
During the past weeks the OSM had massive server problems, cause a lot
of Apps embedded our webmap. For that reason we had to block the most
popular applications [1] cause they simply violate our tiles usage
policy [2].
Now I did some visualisations/compare with the most current logs and
it turns out that OSMand is again under the Top 10 Apps.
http://www.openstreetmap.org/user/!i!/diary/15190
You know our project is about creating a free worldmap (so collecting
geoinformations) and not to deploy a full featured webmap directly for
everyone as Google Maps does. All our server infrastructure is
sponsored and administrated by just a few volunteers, so we have a
lack of every kind of resources to act as a big service.
So can you please use MapQuest or Cloudmade tiles instead or maybe
someone of you can setup an own map rendering stack?
[1] http://wiki.openstreetmap.org/wiki/Blocked_applications
[2] http://wiki.openstreetmap.org/wiki/Tile_usage_policy
bye
Matthias (user:!i!)
andre van atten
!Hi!
I understand the problem of overload of the servers, and there can thought about ideas to reduce this load, but there are some aspects to it I would like to mention.
1. You say OpenStreetmap is about collecting geoinformation and not about deploying it. So you mean it is for the creators and not for the users? However, that is conflicting with the mission of OSM. And it is also conflicting with the way OSM is profiling itself as a free map of the world to anyone. Why does OSM not have a restricted user login then, so you need an account, and this account can be tracked? Or a tile api with account which is needed to request tiles? If not doing so, everyone who uses OSM data considers it as free and for nothing.
2. Osmand does a very good job by REDUCING the load on tile servers, because it delivers vector based maps, which can be pre downloaded from the Google servers. So Osmand should be sponsored by OSM ;-), because Osmand is inventing new ways of OSM data distribution.
3. Why does OSM not deliver sqlite packages of tiles from a separate download server? Many apps can handle these, but everyone has to make its own packages. People use now batch downloading from app or from osmmapcreator, because they can't download them directly. Of course, Osmand can host these sqlite maps, but why should we solve OSM problems if OSM does not solve their own issues too?
4. OSM delivers data in a technically spoken very inefficient way. Either by using old fashioned very small tiles, which request enormous amount of server overhead in connections and caching and http overhead. The other option is xml , file based or xapi based, which takes too much space and time too, and is not for end users. There is no redistributable vector file format for end-users, others then OSM commercial partners deliver, like geofabrik (shp en protobuf) and cloudmade. I think OSM should be thinking about what their mission is, and how to renew and re-invent themselves.
In the mean time Osmand can think about some possible solutions:
1. Restrict the Osmand app in total MB or number of tiles to download. Or restrict tiles of levels 17 and higher.
2. Deliver sqlitedb packages of fixed areas of OSM data,like areas if 10 by 10 km, ready to download.
3. Additional deliver hillshade maps (tiles in sqlitedb format with labeled height contour lines and hillshading) or srtm vector maps with contourlines. Because Osmand can combine layers with opacity settings, they are very useful to reduce the load on hillshade servers Luke OSM cyclemap.
Andre
Yves CAINAUD
All in all, Osm faces technical difficulties and ressource issues that needs to be solved, and apparently for this reason, Osm is data-oriented for a while now.
Clearly, tiles bulk download is *strongly discouraged*, and *forbidden* for zoom 17 and higher, and this for ages.
Yves
!i!
On 30 Okt., 12:23, andre van atten <andrevanat...@gmail.com> wrote:
> 1. You say OpenStreetmap is about collecting geoinformation and not about
> deploying it. So you mean it is for the creators and not for the users?
> However, that is conflicting with the mission of OSM. And it is also
> conflicting with the way OSM is profiling itself as a free map of the world
> to anyone. Why does OSM not have a restricted user login then, so you need
> an account, and this account can be tracked? Or a tile api with account
> which is needed to request tiles? If not doing so, everyone who uses OSM
> data considers it as free and for nothing.
project there is a bunch of spectrum of opinions on what is the
mission of OSM. The Tile usage policy says that you have to provide a
trackable usaer agent, but of course we are open and not willing to
say people, what not to do with our material ;) But naturaly there are
constrains by our infrastructure and we have to make sure that the
project itself can life (and so decouple it from the ecosystem around
it).
> 2. Osmand does a very good job by REDUCING the load on tile servers,
> because it delivers vector based maps, which can be pre downloaded from the
> Google servers. So Osmand should be sponsored by OSM ;-), because Osmand is
> inventing new ways of OSM data distribution.
traffic. Are you sure, you fullfil the policy? (these aren't optinal
but enforced)
> 3. Why does OSM not deliver sqlite packages of tiles from a separate
> download server? Many apps can handle these, but everyone has to make its
> own packages. People use now batch downloading from app or from
> osmmapcreator, because they can't download them directly. Of course, Osmand
> can host these sqlite maps, but why should we solve OSM problems if OSM
> does not solve their own issues too?
to date
http://wiki.openstreetmap.org/wiki/Planet.osm
I'm sure your idea is very welcome but there are so much different
applications (and types as rendering, routing, searching,...) that
have individual requirements and have only a few people involved in
Developing software (but so much users doing pretty cool mapping
actions :) )
> 4. OSM delivers data in a technically spoken very inefficient way. Either
> by using old fashioned very small tiles, which request enormous amount of
> server overhead in connections and caching and http overhead. The other
> option is xml , file based or xapi based, which takes too much space and
> time too, and is not for end users. There is no redistributable vector file
> format for end-users, others then OSM commercial partners deliver, like
> geofabrik (shp en protobuf) and cloudmade. I think OSM should be thinking
> about what their mission is, and how to renew and re-invent themselves.
the toolchain around it?
Well in the german forums some people talking about a vector tile
services but I'm not sure how far they are currently...
>
> In the mean time Osmand can think about some possible solutions:
> 1. Restrict the Osmand app in total MB or number of tiles to download. Or
> restrict tiles of levels 17 and higher.
> 10 km, ready to download.
> height contour lines and hillshading) or srtm vector maps with
> contourlines. Because Osmand can combine layers with opacity settings, they
> are very useful to reduce the load on hillshade servers Luke OSM cyclemap.
a central tile proxy, if you have servers that can do the job.
>
> Andre
> Op 30 okt. 2011 11:05 schreef "!i!" <thecoremaste...@googlemail.com> het
Jindřich Makovička
> In the mean time Osmand can think about some possible solutions:
> 1. Restrict the Osmand app in total MB or number of tiles to download. Or
> restrict tiles of levels 17 and higher.
> 2. Deliver sqlitedb packages of fixed areas of OSM data,like areas if 10 by
> 10 km, ready to download.
> 3. Additional deliver hillshade maps (tiles in sqlitedb format with labeled
> height contour lines and hillshading) or srtm vector maps with contourlines.
> Because Osmand can combine layers with opacity settings, they are very
> useful to reduce the load on hillshade servers Luke OSM cyclemap.
How about changing the default map from OSM/Mapnik to CloudMade or MapQuest?
--
Jindrich Makovicka
Victor Shcherb
andre van atten
Jindrich,
That has no advantage at all, because user can change his default map. Mine is always set to Osmand vector maps of course;-)
By the way, should we give Osmand a different name then?
CloudAnd or QuestAnd ;-) Maybe Cloudmade is willing to sponsor us on that, if we use their rendering scheme for Vector maps?? @Victor: why don't give it a try;-)
I think Osmand users should always be able to use OSM tiles in the normal way. The download area option in Osmand can be restricted to some megabytes, and exclude levels 17 and more. In case of OsmandMapCreator, it should get a different default user agent, that is showing good will. And maybe it is an idea to deliver standard areas if 10*10 sq km OSM tiles as sqlitedb packages. However, that means a lot of maintaining, we could make a bot for that, but then that bot should have its own user agent, and it should get 100% priority and should not work on peak hours, but night hours (what are OSM night hours anyway???)
Andre
Jindřich Makovička
> Jindrich,
>
> That has no advantage at all, because user can change his default map. Mine
> is always set to Osmand vector maps of course;-)
Sure, but a lot of users won't bother changing it, if the default is
good enough. I meant it as an addition to the above measures, not as a
complete solution.
--
Jindrich Makovicka
!i!
On 30 Okt., 13:25, Victor Shcherb <victor.shch...@gmail.com> wrote:
> What I can not understand why it is good point to block any application? If
> user prefer to use application "A" instead of Browser he will use it
> otherwise he goes to browser to see the map. I see point of bunch loader
> make server instable but I suggest different solution :
>
> tiles - 1st min, 500 tiles - for 10 min, 2000 - for 60 min.*
The problem was, that we got to much parallel connections that were
killing the network equipment of the universities that provide our
hosting (for free btw).
I'm not sure what the admins will do, I guess moderating IP filters is
a quite time consuming task, where we havent enough people that would
doing this job?
@All
Why do you think people wouldn't change over to MQ style, if you
explain the situation to them?
Jindřich Makovička
> @All
> Why do you think people wouldn't change over to MQ style, if you
> explain the situation to them?
On the first impression, in non-English speaking countries it's a
clear downgrade, because it strips down the accents/diacritics from
all names on the map.
CloudMade maps look pretty similar to OSM's Mapnik style though.
--
Jindrich Makovicka
andre van atten
Hi,
I think the parallel connections come from pc-based tile pre-downloaders which have options on how many connections are used for downloading. So it helps to work around all downloader apps and restrict them to use 1 connection voluntary. Dont know what Osmandmapcreator uses.
Another option is to configure the OSM load balancer to use only one connectiion per IP address, if that is possible anyway depending on the type of load balancer, or internet server.
Andre
Victor Shcherb
Hi,
Jindřich Makovička
tile.openstreetmap.org.nyud.net instead. But it's slower and I am not
sure how much it would help the upstream servers.
--
Jindrich Makovicka
Yves
I doubt that simple browsing is the real responsible for this traffic.
Yves
--
Envoyé de mon téléphone Android avec K-9 Mail. Excusez la brièveté.
Victor Shcherb <victor....@gmail.com> a écrit :
andre van atten
Pavol Zibrita
Planet file can be updated using diffs. However, I don't know how big it is.....
However, this is only part of the problem, of course.
Regards
Pavol
andre van atten
Hi,
This idea matches with the idea of an pc application that is able to render maps in the same way Osmand does. So instead of running on Android it runs in plain java, so every machine windows, ubuntu, linux , 64bits, can run it. However, it doesn't need gps and navigation, unless one takes his whole laptop in the car (driving) or in the hand (hiking and biking) ;-)
However, I am not joking. It should render on the same obf files, so people can load the protobuf files from geofabrik to build their own obf files, or simply download them from the Google servers.
Our OSM friends would be very happy, and Osmand has the chance to become market leader in offline mapping, because the pc market will grow exponentially, and even more when OSM servers are getting more and more overloaded.
Well, all the traffic is then to the Google code servers to download the obf files, and sure they are willing to help us OSM guys !
@ Victor: make it a Contribution version and you can become rich. Can I have shares in Osmand ;-)?
Andre
Lukas Morlock
our company has a running tileserver with minutely diffs covering central europe, if i could get some administrative help it would be able to be part of such a load-balancing network. please contact me if our help is needed for this superior application.
regards
lukas
Victor Shcherb
Andre,
yes python is ok as long as i dont have to write the script ;)
Everything thats able to run on debian5. I also made some tests regarding the connection limit, i would say we just test it (also single tiles) and limit if needed.
btw: osmand has now some VERY big fans in the philippines and their number will grow because of the nice detailled map coverage and a very poor net coverage on these islands.
Lukas
Am 11/2/2011 9:50 AM, schrieb andre van atten:Great!
We are starting to create some cool solutions!
Concerning the script: it isn't there yet. Sqlite packages can now only be made with Osmandmapcreator. So these scripts should be made.Please read my answer to Victor in the thread "Offline hill shading" this morning.Yves, I see you are very experienced in creating server side scripts. Could you look into it and give us some info about the feasibility of a python script? I suppose sqlite can be integrated in python with not too much effort?
Lukas, I suppose python is OK too?Andre
Op 2 nov. 2011 02:45 schreef "Lukas Morlock" <mor...@frakt.de> het volgende:
Hi Andre
I will start to setup things at the beginnig of next week, first tiles of the entire planet will be available 2 weeks later. one thing that i need from you is a documentation or debian script that processes the single tiles to raster-packages.
Lukas
Am 11/1/2011 5:34 PM, schrieb andre van atten:Lukas,
That looks great. I think for "the common OsmAnd user" a refreshment every 2 - 4 weeks is good enough. If the number of connections can become a problem, I suggest you would not deliver separate tiles, because they use the most overhead on http, bandwidth, internet server and CPU.
Andre
2011/11/1 Lukas Morlock <mor...@frakt.de>
Hi Andre,
i need a little to calculate these tile-statistics, traffic would not be the problem, but maybe the number of connections would slow down other services that we need. Solets start with this, do i understand right that you need the rendered tiles of a world map on our server and a script that generates those raster packages from them? as i mentioned its no problem to render and store the tiles, but its not possible to have the raw data updated to the minute or even week.
besides of that, if you could be able to have bandwith for RASTER packages in sqlitedb format, we could start "caching" tiles in that "offline" format, which would be fantastic. Then some bot could create these packages from the original tiles, so they can be used directly by OSM users. We like to think about an option to download them from within OSmand app. Traffic would not be 5TB, we could start and see what it takes
the server is able to download the complete planet every 2-4 weeks and import them in a fresh database, what takes these 2-4 weeks, so changes in osm-data made after that planet-download will show up 2-4 weeks later.