Google maps custom tile hosting on GAE

[Fred]

Hi,

I'm developing a google maps app, hosted on GAE, for which I've
developed a custom tile overlay, consisting of thousands of png
images, at about 4 - 8k each in size. I've tried uploading about 15mb
worth of tiles (just over 3,500 tiles), and have hit the 1000 file
limit. (Detailed in issue 161 - http://code.google.com/p/googleappengine/issues/detail?id=161
).

The suggestion from google is: "We encourage developers to find
creative ways to reduce the number of files in an app" - but for a
custom overlay in maps I don't think there is any other solution that
to have multiple tiles.

So - my question is whether anyone can suggest a way around this. I
think this is a showstopper for me otherwise.

Thanks!

[Brian Clapper]

You could store them in a zip file, then extract what you need from the zip
file on the fly.
--
-Brian

Brian Clapper, http://www.clapper.org/bmc/
It is a miracle that curiosity survives formal education.
-- Albert Einstein

[Fred]

but "An app can read files, but only files uploaded with the
application code" from http://code.google.com/appengine/docs/whatisgoogleappengine.html

So should the zip file would be imported into the datastore?

On Sep 2, 4:37 pm, Brian Clapper <brian.clap...@gmail.com> wrote:
> On 9/2/08 11:29 AM, Fred wrote:
>
>
>
> > Hi,
>
> > I'm developing a google maps app, hosted on GAE, for which I've
> > developed a custom tile overlay, consisting of thousands of png
> > images, at about 4 - 8k each in size. I've tried uploading about 15mb
> > worth of tiles (just over 3,500 tiles), and have hit the 1000 file

> > limit. (Detailed in issue 161 -http://code.google.com/p/googleappengine/issues/detail?id=161

> > ).
>
> > The suggestion from google is: "We encourage developers to find
> > creative ways to reduce the number of files in an app" - but for a
> > custom overlay in maps I don't think there is any other solution that
> > to have multiple tiles.
>
> > So - my question is whether anyone can suggest a way around this. I
> > think this is a showstopper for me otherwise.
>
> You could store them in a zip file, then extract what you need from the zip
> file on the fly.
> --
> -Brian
>

> Brian Clapper,http://www.clapper.org/bmc/

[Brian Clapper]

On 9/2/08 11:45 AM, Fred wrote:
> but "An app can read files, but only files uploaded with the
> application code" from http://code.google.com/appengine/docs/whatisgoogleappengine.html
>
> So should the zip file would be imported into the datastore?

Your first paragraph should give you the answer. If it contains static data,
you can put it in your local development tree and upload it with appcfg. Or
you can put it in the data store as the equivalent of a BLOB.Brian

Brian Clapper, http://www.clapper.org/bmc/
Boy, n.:
A noise with dirt on it.

[Tim Hoffman]

why aren't you storing your tiles as entities in the store rather than
as files?

T

On Sep 2, 11:29 pm, Fred <fred.leef...@gmail.com> wrote:
> Hi,
>
> I'm developing a google maps app, hosted on GAE, for which I've
> developed a custom tile overlay, consisting of thousands of png
> images, at about 4 - 8k each in size. I've tried uploading about 15mb
> worth of tiles (just over 3,500 tiles), and have hit the 1000 file

> limit. (Detailed in issue 161 -http://code.google.com/p/googleappengine/issues/detail?id=161

[Fred]

OK, thanks Brian. Final question - any pointers for reading from zip
files on the fly?

On Sep 2, 5:02 pm, Brian Clapper <brian.clap...@gmail.com> wrote:
> On 9/2/08 11:45 AM, Fred wrote:
>
> > but "An app can read files, but only files uploaded with the

> > application code" fromhttp://code.google.com/appengine/docs/whatisgoogleappengine.html

>
> > So should the zip file would be imported into the datastore?
>
> Your first paragraph should give you the answer. If it contains static data,
> you can put it in your local development tree and upload it with appcfg. Or
> you can put it in the data store as the equivalent of a BLOB.Brian
>

> Brian Clapper,http://www.clapper.org/bmc/

[Fred]

Tim:

> why aren't you storing your tiles as entities in the store rather than
> as files?

I was concerned about CPU quotas - each tile request would hit the
datastore.

[bFlood]

1) put them in the datastore and memcache the mostly commonly used
(like the tiles at the startup level)
2) set client side cache headers for the tiles if they don't change
often
3) use alternative storage like Amazon S3

personally, i would not try the datastore method first. I have not
seen an issue yet with storing tiles as blobs

cheers
brian

> > > Thanks!- Hide quoted text -
>
> - Show quoted text -

[Fred]

Hi Brian,

> personally, i would not try the datastore method first. I have not
> seen an issue yet with storing tiles as blobs

do you mean you *would* try the datastore method first? How many tiles
are you typically storing in the datastore?

[Brian Clapper]

On 9/2/08 12:06 PM, Fred wrote:
> OK, thanks Brian. Final question - any pointers for reading from zip
> files on the fly?

See the standard Python zipfile module.
http://docs.python.org/lib/module-zipfile.html

Brian Clapper, http://www.clapper.org/bmc/
Variables won't. Constants aren't.

[bFlood]

whoops, little typo: "I would not try" should be "I would try".
there's no limit on the number of blobs you can store but you will hit
the storage quota in the preview. I have thousands in there right now
for testing purposes

It might be cheaper to store them on Amazon S3 but I have not worked
out the difference in price (both charge for storage, bandwidth and
some form of cpu activity, in S3 it;s the GET/PUT costs). I'm going to
wait until GAE is out of preview before really comparing the cost

brian

> > > - Show quoted text -- Hide quoted text -

[Wooble]

On Sep 2, 12:09 pm, Fred <fred.leef...@gmail.com> wrote:

> Tim:
>
> > why aren't you storing your tiles as entities in the store rather than
> > as files?
>
> I was concerned about CPU quotas - each tile request would hit the
> datastore.

If the information in Issue 161 is correct and the reason for the 1000
file limit is that the files themselves are being stored in the
datastore, I imagine you'll get a similar CPU hit whether you're
accessing "files" or pulling them out of the datastore directly, and
I'd also imagine that the CPU time involved in loading a zip file of
thousands of tiles, unarchiving it, and then finding the tile(s) you
want would be even worse.

[Fred]

Thanks all, the verdict then is to try the datastore first. The tiles
will be refreshed on a regular basis (probably every 24 hours) using
an application outside the GAE, so the reliability and speed of the
bulk uploader will be an important factor. The last time I tried the
bulk uploader I found it would fail occassionally - perhaps it's more
reliable now.

Otherwise, I'll give Amazon's S3 a try.

[Alper]

Hi,

I want to ask another thing about tile hosting.

I have been using datastore for storing tiles for more than 2 weeks
and there is no problem about datastore, but the problem is upload
tiles to datastore.

Sample tile set is composed of approximately 5500 tiles with 27 MBs.

I have tried 2 different methods to load :

1.Method : (take 8 minutes to fetch)
I have uploaded all tiles to internet and GAE fetch it from the web.
This method is faster than 2.Method but I don't want to do a second
work as uploading to the internet before fetching.


2.Method: (take 10 hours to POST)

I have uploaded tiles via POST with Curl from my computer but it is
very very slow compared to the 1.Method which can be seen above.

My question is that is there a way to accelerate the uploading via
POST method?

Thanks.

Alper.

[bFlood]

POST - run multiple upload batches at once. each tile is probably
stored independently so there should be no write contention. uploading
10 tiles at the same time will (mostly) give you 10x performance
increase. in Arc2Earth, our bulk tile uploader works on multiple
folders and we have not seen a problem yet

what is the duration of each of your POST uploads? that sounds like a
lot of time for 5500 requests. I'm seeing ~70ms per tile request
(256x256, 5kb-25kb PNGs)

[Fred]

I've been using POST, on a tile by tile bases. My last upload of 7500
tiles, approx 4k - 8k each, took about 40 minutes. My DSL connection
is about 12mb down, 1mb upload. As this number increases I'll try
Brian's suggestion of grouping tiles together with each POST.

[Alper]

Hi Brain,

Sorry for late response, I was on a trip.

I will try multiple upload on same page with multiple uploading
within a week.
I will write the results after finishes.

Thanks.

Alper.