Bad requests on custom tile overlays

[Kesuke]

I am using custom tile overlays on a map. To fetch the tiles I use the
generic method:

function createImageMapType(id, opacityvalue) {
return new google.maps.ImageMapType({
getTileUrl: function(coord, zoom) {
return "/tiles/" + id + "/"+ zoom + "_" + coord.x + "_" +
coord.y + ".png";
},
tileSize: new google.maps.Size(256, 256),
isPng: true,
opacity: opacityvalue
});
}

Here is the problem: My map uses lots of transparent PNG tiles, and
infact some tiles are completely transparent. So rather than have
thousands of 3kb empty PNG files, I just don’t have those tiles.

This works but it technically generates a lot of 404 bad request
errors. An average user visit can make as many as 300 bad requests.
This really hurts page load time and also looks bad for search
engines.

So, does anyone have an idea how I could only fetch tiles that
actually exist?

One other idea I have had is could I setup an .htaccess rule that
redirects to a universal empty PNG file for any tile that can’t be
found (so it doesn’t return a bad request).

I don’t want to generate empty tiles as I have over 130 custom tile
overlays for zoom levels 1 through 5. Just at zoom level 5 that would
be something like 140,000 files and my server is limited.

[Jason]

I'd really like to know a fix for this too. It's been a problem for me
for quite a while and I can't easily get write permission
to .htaccess.

[Martin™]

Why not make all tile requests to a PHP script?

The script can check if the requested tile exists.
If the tile exists then the script can return it to the map, otherwise
it can return a blank transparent tile.

http://php.net/manual/en/function.file-exists.php

http://www.php.net/manual/en/function.file-get-contents.php

A transparent 1 x 1 pixel PNG image can be used where there is no tile
and that could be as small as 95 bytes:

http://code.martinpearman.co.uk/deleteme/transparent_1x1.png

Martin.

[John Coryat]

Martin's suggestion is the standard way of doing it. Make a tile server that does this simple function and you'll notice your server runs faster. Processing a 404 is time consuming.

If you'd like to see a Perl example, see http://www.usnaviguide.com/ws-2008-02 - look for download.zip

-John Coryat

[Martin™]

Just a thought.

Will a browser cache tiles served via a server-side script or not?

Any idea?

Martin.

On Jul 29, 5:43 am, John Coryat <cor...@gmail.com> wrote:
> Martin's suggestion is the standard way of doing it. Make a tile server that
> does this simple function and you'll notice your server runs faster.
> Processing a 404 is time consuming.
>

> If you'd like to see a Perl example, seehttp://www.usnaviguide.com/ws-2008-02- look for download.zip
>
> -John Coryat

[Barry Hunter]

>
> One other idea I have had is could I setup an .htaccess rule that
> redirects to a universal empty PNG file for any tile that can’t be
> found (so it doesn’t return a bad request).

Thats what I do. In my experience its lighter weight than using a
custom handler (say php)

Use the rewrite engine to perform the redirect, rather than a
ErrorDocument, that has too much overhead.

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /img/transparent_1x1.png [L]

[Barry Hunter]

On Fri, Jul 29, 2011 at 11:05 AM, Martin™ <warw...@gmail.com> wrote:
> Just a thought.
>
> Will a browser cache tiles served via a server-side script or not?

It can. Needs the script to set the right caching headers.

At one time many proxies wouldnt cache a request with a ? in it. But
most now do, when they have the proper cache headers. The browser
cache doesnt care about the ?, it would cache it anyway.

>
> Any idea?
>
> Martin.
>
>
> On Jul 29, 5:43 am, John Coryat <cor...@gmail.com> wrote:
>> Martin's suggestion is the standard way of doing it. Make a tile server that
>> does this simple function and you'll notice your server runs faster.
>> Processing a 404 is time consuming.
>>
>> If you'd like to see a Perl example, seehttp://www.usnaviguide.com/ws-2008-02- look for download.zip
>>
>> -John Coryat
>

> --
> You received this message because you are subscribed to the Google Groups "Google Maps JavaScript API v3" group.
> To post to this group, send email to google-map...@googlegroups.com.
> To unsubscribe from this group, send email to google-maps-js-a...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.
>
>

[Jason]

Does anyone know of a page or an example where this is done with PHP?
Makes sense but I haven't done much PHP.

Thanks

[Barry Hunter]

<?php

$filename = "tiles/".intval($_GET['z'])."/".intval($_GET['x'])."/".intval($_GET['y']).".png";

header("Content-Type: image/png");
header("Cache-Control: max-age=84600");

if (file_exists($filename)) {
header("Content-Length: ".filesize($filename));
readfile($filename);
} else {
header("Content-Length: 95");
readfile("transparent1x1.png");
}

Change to suit your needs.

[John Coryat]

One advantage of using a tile server is you can add some security to your tile requests. 

In my experience, anything not protected will eventually be abused by someone or something. Adding a simple layer of security, such as a key code or even a simple algorithm like Google uses, will keep the unauthorized from abusing your bandwidth and intellectual property. Of course, anyone sufficiently motivated will figure out how to bypass any security measure but by doing at least a single level of security, you're indicating you don't want to be abused and that often is enough to keep the problem under control.

-John Coryat

[Jason]

Sorry for the noob question but does this require any other
configuration changes (ie. htaccess)? I've included this in my
index.html and it is still showing 404 errors in Chrome.

Thanks

On Jul 29, 2:19 pm, Barry Hunter <barrybhun...@gmail.com> wrote:
> <?php
>
> $filename = "tiles/".intval($_GET['z'])."/".intval($_GET['x'])."/".intval($_GET['y']).".png";
>
> header("Content-Type: image/png");
> header("Cache-Control: max-age=84600");
>
> if (file_exists($filename)) {
>   header("Content-Length: ".filesize($filename));
>   readfile($filename);} else {
>
>   header("Content-Length: 95");
>   readfile("transparent1x1.png");
>
> }
>
> Change to suit your needs.
>

[John Coryat]

On Friday, July 29, 2011 6:20:03 PM UTC-5, Jason wrote:

Sorry for the noob question but does this require any other
configuration changes (ie. htaccess)? I've included this in my
index.html and it is still showing 404 errors in Chrome.

Thanks

You'll have to be more specific. What did you try? What's the URL of your map? 

-John Coryat 

[Kesuke]

Okay, thanks everyone – the PHP solution works great!
So here is a guide for Jason or anyone else who stumbles across this;

1.) Make a new PHP file with the following code:

<?php

$filename = "/usr/www/httpdocs/tiles/" . intval($_GET['z']) . "_" .
intval($_GET['x']) . "_" . intval($_GET['y']) . ".png";

header("Content-Type: image/png");
header("Cache-Control: max-age=84600");

if (file_exists($filename)) {
header("Content-Length: " . filesize($filename));
readfile($filename);
} else {

readfile("/usr/www/httpdocs/clear.png");
}

?>

Basically this PHP script will check to see if the file exists. If it
does it will return the file. If it doesn't, it will return a 1px
empty image, in this case called 'clear.png'. The only stuff you
should need to change are the paths to clear.png and $filename. Be
aware that PHP safe mode can probably screw this up.

NOTE: counter-intuitively, PHP looks for the true file destination NOT
the URL. So you need to know your servers C:// style path to the file.
There are ways around it using fopen() but you would need to look
those up - not realising this gave me headaches.

2.) Now, go into your map javascript and point to the PHP file instead
of the image URL. Specify the zoom/x/y co-ords which we will detect
using $_GET in our PHP script. If you don’t know what $_GET is have a
look on google as its quite fundamental – but in basic terms you send
variables and their values to a PHP document by appending the URL with
the format “map.html/?var=value&othervar=value”. PHP then plucks those
values and variables out for you to use.

So, using the above example:

function createImageMapType(id) {

return new google.maps.ImageMapType({
getTileUrl: function(coord, zoom) {

return "/tilefinder.php?z=" + zoom + "&x=" + coord.x + "&y=" +
coord.y;

},
tileSize: new google.maps.Size(256, 256),
isPng: true,

opacity: true
});
}

I hope this helps. So far it seems to be working well on my site,
there is a noticable improvement in load times and my Google PageSpeed
score got pushed up by about 20-30 points. If you want to improve
things further, I suggest setting a higher max-age (probably of 28
days/2419200) and also adding a last modified content-type header.

[Kesuke]

Correction, after deleting my firefox cache to see the full benefits
Google PageSpeed has gone from ~45/100 before these changes to
~90/100! Thats pretty solid stuff for a map with 1.5mb of data to load
just on initialize. It has also cut the number of 404s from ~100 on
load down to 0.

Furthermore, the benefits of setting longer max-ages on the files
reduces the actual transfer size for a second-visit down by over 50%
to 650kb. Obviously if you view a new area of the map you have to
download fresh tiles, but on startup it makes a big difference.

The PageSpeed addon did have a slight gripe with me feeding it the
same 1px clear.png image from different URLs but it doesn't seem to
appreciate that is the lesser of two evils. So I set a very long
expiry on the clear.png and PNGcrushed it down to a massive 69 bytes
and PageSpeed seems to be ignoring it now.

> > -John Coryat- Hide quoted text -
>
> - Show quoted text -

[Jason]

Thanks for posting that Kesuke! It would be great to get this working.
Those 404s have been a headache for quite a while.

I've implemented your example on a small test page and I'm not seeing
any tiles. I'm not too familiar with PHP but I did check that safe
mode is "off" with phpinfo.

My php looks like this:

<?php
$filename = "/var/www/merge/tiles/" . intval($_GET['z']) .
"_" .intval($_GET['x']) . "_" . intval($_GET['y']) . ".png";

header("Content-Type: image/png");
header("Cache-Control: max-age=84600");

if (file_exists($filename)) {
header("Content-Length: " . filesize($filename));
readfile($filename);
} else {

readfile("/var/www/merge/tiles/clear.png");
}
?>

the tiles are at:

http://69.164.209.124/merge/tiles/

and the page is here:

http://69.164.209.124/merge/mimagery.html

Any idea what I'm missing here?

Thanks in advance for a steer in the right direction.

[Kesuke]

At a quick glance, have you noticed the $filename is expecting to
output a link in the following format:

$filename = /var/www/merge/tiles/zoom_x_y.png

But your files appear to me to be organised into something like /var/
www/merge/tiles/zoom/x/y.png ?

Take a closer look at the $filename, I suspect you need to change it a
bit (look at the . "_" . parts especially). Let me know how it goes.

[John Coryat]

Your page is generating errors:

Uncaught ReferenceError: initialize is not defined

I suggesting looking at the JavaScript console.

-John Coryat

[Jason]

Got it.. Thanks John.. I was working on it when you hit it probably.

Anyway, I think I've got it solved. Kesuke's PHP code was looking for
tiles named "zoom_x_y.png". My tms is a bit different, zoom and x are
folders and y is the tile png.

Changing the php $filename to:

$filename = "/var/www/merge/tiles/" . intval($_GET['z']) .

"/" .intval($_GET['x']) . "/" . intval($_GET['y']) . ".png";

and then in the map code changing the return statement to:

return "/merge/tiles.php?z=" + zoom + "/&x=" + coord.x + "/&y=" +
(Math.pow(2,zoom)-coord.y-1) + ".png";

seems to work for me. I have a google base layer, tiles and no 404s.

Kesuke, I can't thank you enough for that post. That problem was
bugging me for months!

Thanks much!

[Kesuke]

Your welcome, i'm also very pleased to have this sorted.

On a side note, you don't need the forward slashes in your javascript,
only your PHP. Even though it works it is probably best to remove them
as it might throw up an error in some browsers.

With $_GET, you stick a ? on the end of the URL. Then the variable
name, followed by the equals sign and the value (number/letters etc.).
If you want to add more than one set of values and variables, seperate
them with the ampersand (&) symbol. You need the slashes in the PHP
obviously because you are telling it to look for the next folder.

[Jason]

Thanks for pointing that out. I'm assuming you mean the "/&x=" and "/
&y="? I thought so too but the odd thing is it works fine without the
forward slashes in Firefox or IE but in Chrome the tiles don't load
without having the forward slashes there.

Not sure why that is.

[Kesuke]

Yup you are correct, it should be "?z=" "&x=" "&y="

Hmmm, my guess is it's probably because you have cached some of the
images in chrome while tweaking it. Try dumping your temp. internet
files, then when you revist the page click Ctrl-F5 to do a force
refresh. It is probably because of the header("Cache-Control: max-
age=84600") - which means that for 84600 seconds (ie 24 hours) the
resources should be retrieved from the cache rather than your server.

> > > - Show quoted text -- Hide quoted text -

[Jason]

Yep, that was it.. Working fine now.

Thanks again!

[Harry G. Osoff]

In my attempt to populate Google Maps from a kml file
var kmllayer = new google.maps.KmlLayer

I only seem to be able to get Blue Markers.

I have attempted using the styles
<style  id></style id>
with <styleUrl></styleUrl>

with both reference to the id with #
and full URL in the kml document with no success.

Any suggestions - links

Thanks.

Harry Osoff
WebDev
1537 News

[James McGill]

Can you post a link to your KML file?

Regards,

James

[Harry G. Osoff]

Sure James:

This was what I first started with.
I have been reading the kml specifications for Google Maps
did i read correctly that only 1000 document wide items are allowed.

I tried breaking this into two smaller files still no luck.

Thanks
Harry


KML File
http://1537news.com/share/Ward5TheftPM.kml

Harry

[geoco...@gmail.com]

On Aug 1, 10:48 am, "Harry G. Osoff" <qualityresea...@gmail.com>
wrote:

> Sure James:
> This was what I first started with.
> I have been reading the kml specifications for Google Maps
> did i read correctly that only 1000 document wide items are allowed.
> I tried breaking this into two smaller files still no luck.
> Thanks
> Harry
> KML File

http://1537news.com/share/Ward5TheftPM.kml

http://www.feedvalidator.org/check.cgi?url=http%3A%2F%2F1537news.com%2Fshare%2FWard5TheftPM.kml

I think this is the problem (invalid xml/KML, en-escaped "&"):
line 5799, column 55: XML parsing error: <unknown>:5799:55: not well-
formed (invalid token)
<description>THEFT FINANCIAL ID THEFT:$300 &UNDER STREET
Arrest = N</d ...

-- Larry

> Harry
> On 7/31/11 18:20:58, James McGill wrote:Can you post a link to your KML file?
>
> Regards,
>

> JamesOn Mon, Aug 1, 2011 at 8:15 AM, Harry G. Osoff<quality...@gmail.com>wrote:In my attempt to populate Google Maps from a kml file

[Rossko]

> KML Filehttp://1537news.com/share/Ward5TheftPM.kml

It's bad form to hijack completely unrelated threads, your KML issue
has nothing to do with custom tile overlays.
Better to start a new thread of your own, for effective responses.

There's an unencoded ampersand in your KML

[Harry G. Osoff]

Larry,

Thanks so much for that reminder.
I had forgot all about the feed validator.

Harry