App Engine internal implementations

[Aviad Moreshet]

Hello,

I'm making a research on distributed systems and would like to gather some information about App Engine's underlying implementations, especially in the following topics:

Architecture, Underlying processes, Communication, Synchronization, Consistency and Replication, Fault Tolerance and Security of course.

I've been googling for hours but all the info I found is about the features the system offers instead of what I'm looking for.

Any intel would be appreciated, so don't hesitate.

Thanks for helpers!

[Nick (Cloud Platform Support)]

Hey Aviad,

It sounds like an interesting project! However, I'm not sure how much information (if any) you'll get for a request for proprietary information in a competitive field as this is by posting on a community forum such as this one. I'd suggest getting in touch with sales as a general contact. If you have an academic qualification to do such research or are in any way able to pull strings, it might be through such a contact.

I regret that other than this, the only possible information I can give is to advise that you check out the documentation, which is a treasure trove of information, which often hints at or explains infrastructure, internals, etc. 

Sincerely,

Nick
Cloud Platform Community Support

[Barry Hunter]

A fair bit  is released in papers

http://research.google.com/archive/bigtable.html

http://research.google.com/archive/paxos_made_live.html

http://research.google.com/archive/spanner.html

(not all are used directly by AppEngine, but bitable underlies the datastore for example) 

https://cloud.google.com/appengine/articles/life_of_write

https://cloud.google.com/appengine/docs/python/datastore/indexes?csw=1

http://googleappengine.blogspot.co.uk/2011/01/announcing-high-replication-datastore.html

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/69a37a45-5670-4e96-9a64-26b49e20b1eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick (Cloud Platform Support)]

Great points @barryhunter!

Another resource is the BigQuery book, which has quite a lot of insight into implementation.

On Thursday, April 14, 2016 at 6:00:09 PM UTC-4, barryhunter wrote:

A fair bit  is released in papers

http://research.google.com/archive/bigtable.html

http://research.google.com/archive/paxos_made_live.html

http://research.google.com/archive/spanner.html

(not all are used directly by AppEngine, but bitable underlies the datastore for example) 

https://cloud.google.com/appengine/articles/life_of_write

https://cloud.google.com/appengine/docs/python/datastore/indexes?csw=1

http://googleappengine.blogspot.co.uk/2011/01/announcing-high-replication-datastore.html

On 14 April 2016 at 21:57, Aviad Moreshet <avia...@gmail.com> wrote:

Hello,

I'm making a research on distributed systems and would like to gather some information about App Engine's underlying implementations, especially in the following topics:

Architecture, Underlying processes, Communication, Synchronization, Consistency and Replication, Fault Tolerance and Security of course.

I've been googling for hours but all the info I found is about the features the system offers instead of what I'm looking for.

Any intel would be appreciated, so don't hesitate.

Thanks for helpers!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

[Aviad Moreshet]

Thanks!

בתאריך יום שישי, 15 באפריל 2016 בשעה 01:00:09 UTC+3, מאת barryhunter:

[Aviad Moreshet]

Thanks Nick, sent them a message.

I've been scanning the docs for a while but most of it is about how to properly use App Engine.

Very few materials about the underlyings.

בתאריך יום שישי, 15 באפריל 2016 בשעה 00:36:34 UTC+3, מאת Nick (Cloud Platform Support):

[Jon Parrott]

Much of the public details of our underlying technology can be found as google research publications. Notably, things in the Distributed Systems and Parallel Computing section. Of particular note are the borg, bigtable, megastore, spanner, and paxos papers. 

You can also check out AppScale, which is an open-source reimplementation of GAE.

[David Snyder]

Hello Aviad -

I read your article, https://medium.com/faun/dry-helm-charts-for-micro-services-db3a1d6ecb80, and thought it was very substantive.

I am supporting a program that is developing system following a micro-services based architecture, containerizing code deliveries, using Kubernetes and Helm.

I am very interested in using an umbrella chart to manage the system as a whole and had some questions.

You indicate in your article, that the use of an umbrella chart allows developers to continue to deploy services independently if they would like to as well as enabling deployment and management of the system as a whole.  Can you expand on this?   How does this approach enable one to deploy services independently.

Secondly, developers have asked how an umbrella chart deals with/accomodates the values.yaml files of each specific microservice.  Do you have any insight into this?

a good email for me is:  david_s...@yahoo.com.

Thank you!  VR,

Dave

On Thursday, April 14, 2016 at 4:57:18 PM UTC-4, Aviad Moreshet wrote: