Performance scaling google maps application

[Henri]

Hi,

I'd like to know what are the best strategies and practises to use
when building a google map that has millions of markers.

Performance is vital. I'm not asking for snips of code but simply
strategy that should be implemented in order to maintain high
performance for loading the markers.

E.g. Use fusion tables? Server side clustering vs client side
clustering? KML over lays?

Much appreciated,
Henri

[Rossko]

> I'd like to know what are the best strategies and practises to use
> when building a google map that has millions of markers.

Go back a step and re-examine your needs. A viewer isn't goimg to get
much value out of a map with millions of markers on it, so you could
perhaps think about what you want to actually show at any given time
(perhaps from an available pool of millions of markers)

[Henri Cammiade]

Yes. Obviously it is pointless displaying markers in areas half way around the world to the view port. The DB has millions of locations - what is the best way to build the gmap for high performance of loading initial markers and for loading additional markers on pan/zoom.

Thanks

--
You received this message because you are subscribed to the Google Groups "Google Maps JavaScript API v3" group.
To post to this group, send email to google-map...@googlegroups.com.
To unsubscribe from this group, send email to google-maps-js-a...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.

[Barry Hunter]

Here is a map that does that
http://wales.portal.geographs.org/map.php?view=thumbs

Ok, that particular map is not using millions of markers, but the same
technique works for bigger datasets. (I just dont have any big
datasets using v3 of the maps api)

The biggest issue is deciding which features to show as the 'sample'.
Picking random works reasonably well. But there are more elaborate
ways of doing it. You could perhaps pick the top rated (if your
dataset has that)

[Barry Hunter]

btw, that is only one approach.

Doing server side 'clustering' might be better suited (rather than
just picking a sample). There are various ways to do that.
https://www.google.com/search?q=server+side+clustering+google+maps

Or you can just outsource it
http://geocubes.com/
one of a few sites offering it :)

[Henri Cammiade]

Excellent, thanks for the advise. Probably server side clustering would be
more suitable as all the results need to "be" there. However, what about KML
overlays or Fusion tables? Has anybody had much experience with either of
these and can comment on the performance?

Thanks,
Henri

[Barry Hunter]

There is a limit of 100,000 geocoded rows in a single table.

So that is a fundermental limitation. It will also only render a
limited amount of data in a single tile (500 features?) - so in heavy
populated areas, features wont show.

(also 100Mb of data per table, and 250Mb per account. Which makes
working with large datasets hard.)

Also there isn't 'clustering' as such. It just tries to plot all
features. Although you do have the heatmap option, which is a sort of
clustering - but does end up obscuring the map with large datasets.

- I would say Fusion tables, is for small or medium datasets - not
large ones - currently.

(I use geocubes - with a dataset of 2.1M features, so that does scale
to large datasets! Note I am only a user, not an affiliate!)

And KML definitly wont cope with this - at least on Google Maps (1000
features is a practical limit - but can do slightly more). KML can
however be used for Google Earth - as can use regions to make a
SuperOverlay/SuperLayer.

[Henri Cammiade]

From what I read about fusion tables, up to 100k rows can be marked on the
map. However, if your fusion table has say 1m rows, you are able to query
the relevant rows required to be mapped using the fusion api and then plot
from the views table.

So, if there were 1m rows for data points across the whole of the UK, but
only 50k in Cardiff then you would be able to select from the 1m rows to
create a view table with 50k rows then you would be able to display these on
the map.

https://groups.google.com/forum/?fromgroups#!topic/fusion-tables-users-group/385ig9FG3UM

Is that what Rebecca from this thread is suggesting or have I misread?
Because I've tried uploading 1.3 million rows to the fusion tables and its
well within the 100mb limit.

Thanks

[davie]

Hi
From reading the relevant constraint ie
Queries with spatial predicates only return data from within this first 100,000 rows. Therefore, if you apply a filter to a very large table and the filter matches data in rows after the first 100K, these rows are not displayed.
The maximum size of table you can filter(Query) is 100K.

Regards Davie

> To: google-maps-js-api-v3@googlegroups.com

>>>> google-maps-js-api-v3@googlegroups.com.

>>>> To unsubscribe from this group, send email to

>>>> google-maps-js-api-v3+unsub...@googlegroups.com.

>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-maps-js-api-v3?hl=en.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups
>>> "Google Maps JavaScript API v3" group.
>>> To post to this group, send email to

>>> google-maps-js-api-v3@googlegroups.com.

>>> To unsubscribe from this group, send email to

>>> google-maps-js-api-v3+unsub...@googlegroups.com.

>>> For more options, visit this group at
>>> http://groups.google.com/group/google-maps-js-api-v3?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google Maps JavaScript API v3" group.
> To post to this group, send email to

> google-maps-js-api-v3@googlegroups.com.

> To unsubscribe from this group, send email to

> google-maps-js-api-v3+unsub...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/google-maps-js-api-v3?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google Maps JavaScript API v3" group.
> To post to this group, send email to

> google-maps-js-api-v3@googlegroups.com.

> To unsubscribe from this group, send email to

> google-maps-js-api-v3+unsub...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/google-maps-js-api-v3?hl=en.
>

--
You received this message because you are subscribed to the Google Groups
"Google Maps JavaScript API v3" group.

To post to this group, send email to google-maps-js-api-v3@googlegroups.com.

To unsubscribe from this group, send email to

google-maps-js-api-v3+unsub...@googlegroups.com.

[Henri Cammiade]

Hi,

 

Yes I read that too. However, in the post Rebecca seems to suggest otherwise:

 

https://groups.google.com/forum/?fromgroups#!topic/fusion-tables-users-group/385ig9FG3UM

 

Do you agree?

 

Thanks

To view this discussion on the web visit https://groups.google.com/d/msg/google-maps-js-api-v3/-/V7cxfZ6GVQgJ.
To post to this group, send email to google-map...@googlegroups.com.
To unsubscribe from this group, send email to google-maps-js-a...@googlegroups.com.

[davie]

Hi
No

	Nope, it's not the size of the result set after you make a query, but the number of rows in the first place. Assume your table is 200K and the required filter,view,query data is all after 100k the filter,view,query data result will be zero Regards Davie

To post to this group, send email to google-maps-js-api-v3@googlegroups.com.

To unsubscribe from this group, send email to google-maps-js-api-v3+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.

To post to this group, send email to google-maps-js-api-v3@googlegroups.com.

To unsubscribe from this group, send email to google-maps-js-api-v3+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.

To post to this group, send email to google-maps-js-api-v3@googlegroups.com.

To unsubscribe from this group, send email to google-maps-js-api-v3+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.

To post to this group, send email to google-maps-js-api-v3@googlegroups.com.

To unsubscribe from this group, send email to google-maps-js-api-v3+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.

To post to this group, send email to google-maps-js-api-v3@googlegroups.com.

To unsubscribe from this group, send email to google-maps-js-api-v3+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.

To post to this group, send email to google-maps-js-api-v3@googlegroups.com.

[Barry Hunter]

Yes exactly :)

But it can be "worked around" - using View Tables. Creating a new table filtering required rows from the first. 

So the 'filter' definition, moves from being a query - to being used to create a new table. Dont need to use a filter on this second table. 

The problem with this approach, is creating a view is not 'instant' in my experience. The table is created. But if you try to use it too soon, it doesnt contain any data. 

So its not suitable for 'dynamic' displays. (eg allowing the user to select a point - then providing a map of that location) - the other issue will need maintainance to delete the views afterwards - otherwise risk depleting quota. 

But if you can predefine a smallist number of view tables, and use them. (can even display 5 FT tables at once on a map) Then it might work. 

FT still doesn't provide clustering tho!

To view this discussion on the web visit https://groups.google.com/d/msg/google-maps-js-api-v3/-/AHE-VlbIp1sJ.

To post to this group, send email to google-map...@googlegroups.com.
To unsubscribe from this group, send email to google-maps-js-a...@googlegroups.com.

[Henri Cammiade]

Hi Barry,

 

Thanks for the reply - that's what I was trying to understand about the View tables using the Fusion api. I presume each time a View table gets created, it simply adds another table to your account quota (up to five tables)?

 

I have a database with over 1 million data points. 1 million markers would NEVER be needed to be mapped at one time. I'm seeking for a fast, high performance solution to a 'dynamic' display as you previously mentioned, where the user selects a location that the map centres and zooms on, then to grab the relevant results from the 1 million to then plot the markers. At most, only around 10k markers would ever be needed to be mapped.

 

I guess Fusion tables are out for doing this? What would I need to do to achieve this?

 

Thanks

To unsubscribe from this group, send email to mailto:google-maps-js-api-v3%2Bunsu...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google Maps JavaScript API v3" group.

[Barry Hunter]

On Tue, Mar 6, 2012 at 12:47 PM, Henri Cammiade <henri.c...@gmail.com> wrote:

Hi Barry,

 

Thanks for the reply - that's what I was trying to understand about the View tables using the Fusion api. I presume each time a View table gets created, it simply adds another table to your account quota (up to five tables)?

The "5" - is now many tables you can display at once on a map. A FT account can contain many many tables. I dont know the formal limit, but suspect 1000

So just displaying 1 table at a time - can have many. 

The quota to worry about is the 250Mb per account. Each View will consume quota - as it in effect takes a copy of the data. (its a materialized view, from traditional database theory) 

 

 

I have a database with over 1 million data points. 1 million markers would NEVER be needed to be mapped at one time. I'm seeking for a fast, high performance solution to a 'dynamic' display as you previously mentioned, where the user selects a location that the map centres and zooms on, then to grab the relevant results from the 1 million to then plot the markers. At most, only around 10k markers would ever be needed to be mapped.

This will work. You either need to 

1. Build a solution for checking when the table is ready. So the user selects the point - and then has to wait while their personal Fusion Table is built. Once the user has finished, need to arrange to the table to be deleted. 

or 2. Build 'canned' tables. A series of tables - predefined ready for use. So when the user selects a point, you just pick the approiate table. Maybe use a grid system? Need to keep track of the tables of course!

 

I still have reservations that FT is suitable for this tho - from a user perspective. When zoomed out, they will see a map with 10,000 points - have to be very careful to keep that map useable. If the whole map is obscured in points, then can't see the map to zoom in! (although say an overview map may help with this) 

[Henri Cammiade]

Thanks for the reply, Barry.

 

From what you’re saying, Fusion Tables don’t seem like a feasible solution yet. Especially, if the user will have to wait for their View Table to get created each time they do a search.

 

I’ve had two major problems – Long delays after map loads until markers get plotted, and poor map performance after markers have been plotted (even with client side clustering).

 

So, from my dataset I’m really just looking the best method to quickly grab the required data from the database and to have them plotted quickly. At most, around 10k markers will be within the view port, so to have high performance when panning/zooming.

 

I saw this example using the Flash Maps api – in terms of map performance with lots of markers it seemed really good...but the Flash Maps api is now deprecated

 

http://www.scribblemaps.com/markerComplex/

 

How would you tackle this?

 

Thanks

 

From: Barry Hunter

Sent: Tuesday, March 06, 2012 12:57 PM

To: google-map...@googlegroups.com

Subject: Re: [Google Maps API v3] Re: Performance scaling google maps application

 

[Barry Hunter]

> How would you tackle this?

Server side clustering.

Either will have to build or find a solution for your particular
database/language of choice

Or outsource it to the likes of geocubes. Or Google Builder may well do it
http://www.google.com/enterprise/earthmaps/builder.html
.. have no experience myself.