Geospatial/geolocation queries from GAE (or other Cloud products)?

1,396 views
Skip to first unread message

Alex Kerr

unread,
Jan 4, 2016, 3:50:44 PM1/4/16
to Google App Engine
Hi all,
Wondering how best to implement some basic geolocation stuff using Google Cloud products (and open source if appropriate)

I need to search user records I'm storing in Datastore, which include the user's address (in ordinary human readable text format) to see which other users are within a certain distance from them. Seems I can't do a geospatial search directly on Datastore records but found something that suggests a clever little mashup with Google's Search API: http://ralphbarbagallo.com/2013/05/14/app-engine-geospatial-datastore-search-a-new-way/

Questions:
1.) Is there any better (/more standard) way than what's suggested at that link? (I know Cloud SQL allows geospatial searches but ideally I want to use Datastore if possible to store my data in)
2.) Geospatial matches (at least using above techniques) need latitude and longitude - are there any Google APIs or open source PHP stuff I could use to convert addresses into lat/lon?

I'm running on PHP AppEngine by the way and need a solution I can implement very quickly and easily if at all possible!

Many thanks for any info/suggestions!

Adam (Cloud Platform Support)

unread,
Jan 4, 2016, 4:46:45 PM1/4/16
to Google App Engine
The Google Maps Geocoding API is used specifically for converting addresses into geographic coordinates. Once you have the geocode points you can use something like the Haversine formula to compute distance between them. I'm not an expert on the Maps API but I don't believe there is a way to compute distance between addresses directly without first retrieving the lat and long.

Barry Hunter

unread,
Jan 4, 2016, 5:14:44 PM1/4/16
to google-appengine

 
I don't believe there is a way to compute distance between addresses directly without first retrieving the lat and long.

Well can use the Directions API, it will give a driving (or walking etc!) distance between two addresses. Or the Direction Matrix API to get distances between multiple addresses. But it wont really 'scale' to running more than a few tens of calculations. So using the streightline distance is usually used, and that can be done in the application. 

... but before using the Maps APIs terms, read the terms very carefully, neither the geocoder nor the directions API can be used without displaying a Google Map on the website. 





There also is 
but its java only, not even the Cloud DataStore, so not PHP. 

 
I beleive this concept:
might would with the Cloud Datastore, not tried. 
(would need reimplementing in PHP)

 
On Monday, January 4, 2016 at 3:50:44 PM UTC-5, Alex Kerr wrote:
Hi all,
Wondering how best to implement some basic geolocation stuff using Google Cloud products (and open source if appropriate)

I need to search user records I'm storing in Datastore, which include the user's address (in ordinary human readable text format) to see which other users are within a certain distance from them. Seems I can't do a geospatial search directly on Datastore records but found something that suggests a clever little mashup with Google's Search API: http://ralphbarbagallo.com/2013/05/14/app-engine-geospatial-datastore-search-a-new-way/

Questions:
1.) Is there any better (/more standard) way than what's suggested at that link? (I know Cloud SQL allows geospatial searches but ideally I want to use Datastore if possible to store my data in)
2.) Geospatial matches (at least using above techniques) need latitude and longitude - are there any Google APIs or open source PHP stuff I could use to convert addresses into lat/lon?

I'm running on PHP AppEngine by the way and need a solution I can implement very quickly and easily if at all possible!

Many thanks for any info/suggestions!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
 To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7aea1d41-e9a0-40aa-8b22-8d582f63376d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Tom Walder

unread,
Jan 5, 2016, 2:26:29 PM1/5/16
to Google App Engine
Hi Alex,

You can do all of this on App Engine, with PHP using the native Search API.

I usually keep my transactional data in Datastore and then the "searchy" stuff, including Geo search in the Search API (when I'm, not using Elasticsearch).

You will need one or more of these open source libraries (which I wrote [disclaimer)) - They are using the native Google Protocol Buffers API, same as the Python/Java/Go runtimes, so they are super quick.

For Datastore

For Search API - there are specific examples and helper methods for Geo.

Some more reading

Enjoy!

Tom

Jim

unread,
Jan 5, 2016, 2:34:35 PM1/5/16
to Google App Engine
Our needs involve displaying embedded maps in our app with location markers overlayed on the map.  A given app can have 50,000 or more markers (every house in a town) so we can't simply display them all at once.  So, we use a tile-based approach that renders only (roughly) the tiles that are "visible" based on the zoom level of the embedded map.

To accomplish this with Datastore we use the Geohash algorithm:  https://en.wikipedia.org/wiki/Geohash

Geohash converts lat/lon coordinates into a hash, and by removing the rightmost digits from the hash you can get bounding rectangles of less granularity as you remove more characters from the right of the hash.  In this way you can zoom in and out with the map and issue simple Datastore queries to retrieve all the records with the matching hash portion.  To further improve end-user performance we also pre-render the different "tiles" and store them as blobs in GCS.

It isn't pretty like the geospatial queries you could do with Cloud SQL, but it works just fine and performs really well.

Alex Kerr

unread,
Jan 6, 2016, 8:54:11 AM1/6/16
to Google App Engine
Thanks Tom, I much appreciate these libraries which I already planned to use, but wasn't quite sure how to layer my requirements on top of them - I think the primary issue is how to have many records of people that I can manipulate easily (e.g. in DataStore, Search API is a bad match for this I think?) and also work out the nearest other people to them based on a human readable address (or postcode at least). Do you recommend or use any particular methods for Geocoding? As mentioned by barryhunter above use of the Maps APIs requires display of, and results put on, a Google Map, which is not necessarily what I want. These guys look like a potential alternative tho: http://geocoder.opencagedata.com/

Thanks again,
Alex

Tom Walder

unread,
Jan 7, 2016, 4:35:01 AM1/7/16
to Google App Engine
Hi Alex,

Geocoding options are many.

I use the Royal Mail PAF data (commercial). Also, there are options like


Which I have not used, but looks pretty interesting.

There is no restriction on the volume of data you put into the Search API - although you will start to pay if you index or store over the free quota.


Best of luck!

Adam (Cloud Platform Support)

unread,
Jan 8, 2016, 6:29:59 PM1/8/16
to Google App Engine
Yep, there is the Directions API but as you mentioned its not really suited to this use case.


On Monday, January 4, 2016 at 5:14:44 PM UTC-5, barryhunter wrote:

 
I don't believe there is a way to compute distance between addresses directly without first retrieving the lat and long.

Well can use the Directions API, it will give a driving (or walking etc!) distance between two addresses. Or the Direction Matrix API to get distances between multiple addresses. But it wont really 'scale' to running more than a few tens of calculations. So using the streightline distance is usually used, and that can be done in the application. 

... but before using the Maps APIs terms, read the terms very carefully, neither the geocoder nor the directions API can be used without displaying a Google Map on the website. 





There also is 
but its java only, not even the Cloud DataStore, so not PHP. 

 
I beleive this concept:
might would with the Cloud Datastore, not tried. 
(would need reimplementing in PHP)

 
On Monday, January 4, 2016 at 3:50:44 PM UTC-5, Alex Kerr wrote:
Hi all,
Wondering how best to implement some basic geolocation stuff using Google Cloud products (and open source if appropriate)

I need to search user records I'm storing in Datastore, which include the user's address (in ordinary human readable text format) to see which other users are within a certain distance from them. Seems I can't do a geospatial search directly on Datastore records but found something that suggests a clever little mashup with Google's Search API: http://ralphbarbagallo.com/2013/05/14/app-engine-geospatial-datastore-search-a-new-way/

Questions:
1.) Is there any better (/more standard) way than what's suggested at that link? (I know Cloud SQL allows geospatial searches but ideally I want to use Datastore if possible to store my data in)
2.) Geospatial matches (at least using above techniques) need latitude and longitude - are there any Google APIs or open source PHP stuff I could use to convert addresses into lat/lon?

I'm running on PHP AppEngine by the way and need a solution I can implement very quickly and easily if at all possible!

Many thanks for any info/suggestions!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Jeff Schnitzer

unread,
Jan 10, 2016, 2:45:24 PM1/10/16
to Google App Engine
The datastore now supports geospatial indexes directly:


It's alpha, you have to ask for an invite, and I have no idea what the PHP bindings would look like, but it lets you do a "show records within a circle (or box) of a point" queries - which sounds like what you are asking for.

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/80be5c2f-ae67-4ccc-825a-29b68fb6315a%40googlegroups.com.
Message has been deleted

Ludovic Borie

unread,
May 10, 2016, 4:22:31 PM5/10/16
to Google App Engine, je...@infohazard.org
Hi all !

Any update on this topic ? This is a feature that could help me a lot !

Regards,
Ludovic Borie

Le mercredi 20 janvier 2016 06:49:15 UTC+1, Sandeep Dhameshia a écrit :
How do I ask for an invite? I did not find any option.

Manikanta Kondeti

unread,
Jul 15, 2016, 2:46:12 AM7/15/16
to Google App Engine

Hi,

Any update on this topic? 

 I have same problem that is defined in the root post.
Alex, could you post references that are helpful to you?

Thanks,
Manikanta 
Reply all
Reply to author
Forward
0 new messages