App Engine VM-based Backends - Trusted Tester Sign-up

[Takashi Matsuo]

Fellow App Engine Gurus,

We're happy to announce the next generation of App Engine Managed Backends. These Backends utilize the App Engine VM Runtime, allowing developers to run Backends on Compute Engine VMs. By building on top of Compute Engine VMs, developers can:

• take advantage of higher CPU and memory
  
• rely on longer-lived processes
  
• utilize a local filesystem
  
• communicate via native network stacks
  
• execute external processes
  
• access the entire JRE
  
• upload arbitrary Python extensions
  

Given that these are App Engine Backends, you can still use all the App Engine APIs to access the existing managed services (Datastore, Task Queues, Memcache, etc.)

Updating existing Backends to run on Compute Engine VMs is a simple config change:

app.yaml

--------

application: app-id

version: v1

runtime: python27

vm: true

manual_scaling:

  instances: 1

That’s all you need to get started. We’ll pick a deafult VM machine type and spin everything up your behalf. Of course there other options that you can set (including machine type) and these are documented in the Getting Started Guide.

In order to build a great product, we need quality feedback from brave early adopters. If you’re interested test driving, please sign-up at the link below and we’ll take care of the rest.

App Engine VM-based VM Runtime - Trusted Tester Sign-up

Also, if you have any questions, please feel free to send an email to:

appengine-...@googelgroups.com

Thanks!

--

Takashi Matsuo | Developers Programs Engineer | tma...@google.com

[pdknsk]

Wow, this sounds quite fantastic if it works how I imagine it does. Hopefully pricing is oriented more towards current GCE prices.

[Ray]

Will it cost like Compute Engine or it will cost like App Engine? (e.g. how much are the memory?)

[timh]

Thats pretty cool.

Just looked at the SQlite example for backend VM on python.  It raises an obvious question, what app server runtime are you running on the vm backend to support a deployed python app.

Is it the new dev_appserver or something else ?  What sort of concurrency does it support ?

I note in the document the getting started guide all requests pass straight through to the backend, and there for concurrency has to be managed on the vm.  How would someone approach this ?

Thanks

Tim

On Friday, June 21, 2013 3:21:06 AM UTC+8, Takashi Matsuo (Google) wrote:

[Takashi Matsuo]

Hi Ray,

Re: cost

We're still discussing the cost, but the final shape of tariff won't be very different from what you expect I think.

Re: memory

You can choose from a subset list of compute engine instance types.

-- Takashi

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jeff Schnitzer]

I just want to say WOW, this is amazing. I have long grumbled here that backends, in their current form, are pretty useless. But this could change that completely.

I also want to say that ever since Google announced Compute Engine and started focusing on it at I/O, I've had the lingering fear that this represents a reduced commitment to GAE within Google. Hearing about tighter integration like this really gets me excited for the future of GAE.

Thank you GAE team,

Jeff

[Takashi Matsuo]

Hi Tim,

Yes it's cool, really. The reason I segment the sqlite guestbook into a single instance is that there is no easy way to share the sqlite table on the local disks between the instances. We run a multi-threaded server on each instance. If you set threadsafe:true in the app.yaml file, the actual server will run in the muti-threaded mode, so it can handle concurrent requests.

-- Takashi

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[pdknsk]

How does this work on multi-core VMs I wonder. As far as I understand, multi-threading here doesn't mean multi-processing, so each backend still only runs on a single core.

[pdknsk]

PS. It seems like for every great news, Google has sad news in store. It appears that today Google has removed the option to revert to the old Google Groups format. Oh well. Totally unrelated, I know.

[Vinny P]

On Thursday, June 20, 2013 2:21:06 PM UTC-5, Takashi Matsuo (Google) wrote:

Fellow App Engine Gurus,

We're happy to announce the next generation of App Engine Managed Backends. These Backends utilize the App Engine VM Runtime, allowing developers to run Backends on Compute Engine VMs. By building on top of Compute Engine VMs, developers can:

• access the entire JRE
  
• upload arbitrary Python extensions
  

+1 to being able to access the entire JRE! Now I can get access to java.awt.* for graphing libraries.

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

[Chris Ramsdale]

re: multi-core, you can os.fork() in Python and ForkJoinPools in Java. 

re: Google Groups, we can only do so much :)

-- Chris

On Fri, Jun 21, 2013 at 12:58 PM, pdknsk <pdk...@gmail.com> wrote:

How does this work on multi-core VMs I wonder. As far as I understand, multi-threading here doesn't mean multi-processing, so each backend still only runs on a single core.

--

[Vinny P]

On Fri, Jun 21, 2013 at 2:58 PM, pdknsk <pdk...@gmail.com> wrote:

How does this work on multi-core VMs I wonder. As far as I understand, multi-threading here doesn't mean multi-processing, so each backend still only runs on a single core.

 

Be careful when using multiple cores within Java applications; it may be faster to run everything on a single core. The author of Mailinator sped up his server by 6x by moving to a single core: http://mailinator.blogspot.com/2010/02/how-i-sped-up-my-server-by-factor-of-6.html

 

The TLDR of the article is that the extra cores cause memory contention depending on the type of CPU used. It might be more cost-efficient to kick up multiple 1-core Compute Engine machines than using a single multicore machine.

 

(I need to test this when I have spare time.)

 

[Nicolas Fonrose (Teevity)]

That really is a great news !

And it is something that makes AppEngine so much simpler to use in situation where we have to mix between PaaS and IaaS ! 

I've got one question regarding the billing :

• I assume this service will be billed as part of the AppEngine bill, right ?
  
• Can you confirm that this will appear as a new line in the AppEngine bill ? 
  

Nicolas

Founder | Teevity | Cloud costs analytics

PS : Now that we've got these great new backends and all the goodness of BigQuery, CloudSQL, Datastore, ..., the only remaining "evil problem" in the AppEngine world is the behavior of the dispatcher which sends "user-facing requests to starting instances".

When this is fixed, it will be near perfection :-) So please, tell us you are going to fix this ...

[Chris Ramsdale]

some comments inline below.

-- Chris

On Mon, Jun 24, 2013 at 7:59 AM, Nicolas Fonrose (Teevity) <nicolas...@teevity.com> wrote:

That really is a great news !

And it is something that makes AppEngine so much simpler to use in situation where we have to mix between PaaS and IaaS!

great to hear, and i'd love to better understand the use cases where this integration helps.  feel free to reach out directly.

 

 

I've got one question regarding the billing :

• I assume this service will be billed as part of the AppEngine bill, right ?
  

still TBD, but there is some design work around this flow.

• Can you confirm that this will appear as a new line in the AppEngine bill ? 
  

ditto to the comment above.

Nicolas

Founder | Teevity | Cloud costs analytics

PS : Now that we've got these great new backends and all the goodness of BigQuery, CloudSQL, Datastore, ..., the only remaining "evil problem" in the AppEngine world is the behavior of the dispatcher which sends "user-facing requests to starting instances".

we have engineers dedicated to fixing this.  once we have an ETA that we're more confident, we'll update the larger group.

 

--

[jon]

That is an absolutely positively excellent piece of news, Chris.

[Sameer Lodha]

This is great news.

The biggest issues we have had thus far has been limited amount of memory (RAM) both with front & back-ends. Although 90% of our process goes thru fine (with the amount of RAM available). There are quite a few instances where more memory is absolutely essential. This should address that pain point effectively.

I just hope, that the management of these backends (provisioning, auto start & auto stop) as simple as the existing (dynamic) backends.

Appreciate the great work that the Appengine team is doing.

--

[Kristopher Giesing]

PS : Now that we've got these great new backends and all the goodness of BigQuery, CloudSQL, Datastore, ..., the only remaining "evil problem" in the AppEngine world is the behavior of the dispatcher which sends "user-facing requests to starting instances".

we have engineers dedicated to fixing this.  once we have an ETA that we're more confident, we'll update the larger group.

That is absolutely fantastic news! 

[Robert Kluin]

This is a really exciting feature.

Will the durability of these backends look and feel more like GCE (long-term / stable) or will the be more like GAE's current backends (flaky and unstable)?

Robert

On Thursday, June 20, 2013 2:21:06 PM UTC-5, Takashi Matsuo (Google) wrote:

[Takashi Matsuo]

Hi Robert,

The VMs won't live forever, but definitely live long enough for the most use-case. Generally there will be one periodic restart per weak.

-- Takashi

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Vinny P]

 +1, this is item #1 on my Christmas list!

[Andrew First]

This is great! Any plans to support autoscaling as well?

Andrew

[Chris Ramsdale]

Manual scaling for a V1 launch, but we're looking into auto-scaling.

-- Chris

--

[Marcel Manz]

Will those VM-based backends support all runtimes? So Java, Python and PHP?

-- Marcel

[Takashi Matsuo]

Hi Marcel,

Currently VM-based backends support Java and Python2.7.

I'd appreciate it if you could file issues for supporting PHP and Go at:

https://code.google.com/p/googleappengine/issues/list?can=2&q=Component%3DVMRuntime

I'd like to know how many people have interests in it.

Thanks for asking!

-- Takashi

On Tue, Aug 27, 2013 at 3:37 AM, Marcel Manz <marce...@gmail.com> wrote:

Will those VM-based backends support all runtimes? So Java, Python and PHP?

-- Marcel

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[Jeffery Fernandez]

Hi Takashi,

Could you explain the differences between appengine backends and VM backends? I am assuming VM backends don't have many restrictions as what appengine does? Could you point me to some documentation or comparison list? Thanks

Mobile: +61 430 534 464

Home: +61 3 9764 4439

Work: +61 3 9723 9399

[Takashi Matsuo]

Hi Jeffery,

For restrictions, you are right. VM-based backends have only fewer restrictions than the traditional App Engine backends.

Currently there's no public documentation. I hope these bullet points help you understanding the difference.

• take advantage of higher CPU and memory
  
• rely on longer-lived processes
  
• utilize a local filesystem
  
• communicate via native network stacks
  
• execute external processes
  
• access the entire JRE
  
• upload arbitrary Python extensions

[Joshua Turmel]

What other options can you specify? This post points to the documentation, the documentation just mentions other options can be set but doesn't say what they are, unless I'm missing something.

Thank you,

Josh

On Thursday, June 20, 2013 2:21:06 PM UTC-5, Takashi Matsuo (Google) wrote:

Fellow App Engine Gurus,

We're happy to announce the next generation of App Engine Managed Backends. These Backends utilize the App Engine VM Runtime, allowing developers to run Backends on Compute Engine VMs. By building on top of Compute Engine VMs, developers can:

• take advantage of higher CPU and memory
  
• rely on longer-lived processes
  
• utilize a local filesystem
  
• communicate via native network stacks
  
• execute external processes
  
• access the entire JRE
  
• upload arbitrary Python extensions
  

Given that these are App Engine Backends, you can still use all the App Engine APIs to access the existing managed services (Datastore, Task Queues, Memcache, etc.)

Updating existing Backends to run on Compute Engine VMs is a simple config change:

app.yaml

--------

application: app-id

version: v1

runtime: python27

vm: true

manual_scaling:

  instances: 1

That’s all you need to get started. We’ll pick a deafult VM machine type and spin everything up your behalf. Of course there other options that you can set (including machine type) and these are documented in the Getting Started Guide.

In order to build a great product, we need quality feedback from brave early adopters. If you’re interested test driving, please sign-up at the link below and we’ll take care of the rest.

App Engine VM-based VM Runtime - Trusted Tester Sign-up

Also, if you have any questions, please feel free to send an email to:

appengine-...@googelgroups.com

Thanks!

[Jeffery Fernandez]

All PHP project owners, please star https://code.google.com/p/googleappengine/issues/detail?id=9953 for introducing support for PHP VMRuntime. Thanks

[Brian Tosch]

I opened an issue for adding Go support.

https://code.google.com/p/googleappengine/issues/detail?id=9964

[Brian Tosch]

Sorry, missed it in my search but there was already an open issue 9950.  I indicated mine is a dupe but I don't think I can retract it.

[Takashi Matsuo]

Thanks Brian for letting us know.

[Mihai]

This allows us to run websockets !!!! however without auto-scaling is worthless .

Thanks,
Mihai.

[Hartwig Erfurth]

Hi, I read that there is Java servlet 3.0 support? Could you give some more hints on this? In which container (Jetty 8?) is it actually based? What EL version is supported? Is CDI (weld) possible? Thanks!

[AK2: A.-"Karriem" A. Khan]

dart vm's 2 ?

[pdknsk]

"Google Compute Engine is now Generally Available"

http://googlecloudplatform.blogspot.de/2013/12/google-compute-engine-is-now-generally-available.html

Hurry! :)

[John Smith]

Thanks for the link it is really helpful for linux user 

[pdknsk]

Any news on general availability? I can't shake the impression of the App Engine team being dormant.

[Amy Unruh]

On 4 February 2014 06:25, pdknsk <pdk...@gmail.com> wrote:

Any news on general availability? I can't shake the impression of the App Engine team being dormant.

As usual, we don't specifically announce timelines ahead of time, but there is an active group of early testers for the VM Runtime-- those who are interested can sign up here:

 http://goo.gl/IjWhtq

(We're soon going to start onboarding a larger group).

[Grant Jason]

Your narrative on VM-based Backends is half-baked without you commenting on the short-comings of current Backends, and how those are solved by this move.

[Tapir]

Could you provide datastore APIs for using on GCE without a GAE instance?

It would be great if it is possible.

[pdknsk]

https://developers.google.com/cloud/managed-vms

[Wesley C (Google)]

@pdknsk: Correct, this feature has been announced publicly now so thanks for sending a link to the developer docs. To see a summary of all the exciting announcements we made today (2014 Mar 25), we suggest you check the blogpost (http://googlecloudplatform.blogspot.com/2014/03/google-cloud-platform-live-blending-iaas-and-paas-moores-law-for-the-cloud.html) and go to http://cloud.google.com for more information.

@tapir: Last year at Google I/O 2013, we launched (http://googledevelopers.blogspot.com/2013/05/get-started-with-google-cloud-datastore.html) the Google Cloud Datastore, which is exactly what you're asking for... access to the Google App Engine datastore without the need for App Engine, accessible from Compute Engine or from anywhere else with an API. The docs can be found at http://developers.google.com/datastore.

Keep an eye out on our Google Developers G+ page (https://plus.google.com/+GoogleDevelopers/posts) for continuous updates on exciting new features we announce on this channel for our users! We've also recently recorded several videos introducing the different bits of the Google Cloud Platform so developers can learn more today! You can find them on our G+ page or watch en masse from the YouTube playlist (https://www.youtube.com/playlist?list=PLOU2XLYxmsIIGo6tf3yFhif9VUSzxjYUT).

Cheers,

-- Wesley

+WesleyChun / @wescpy | Google Developer Relations | wesc...@google.com | Core Python

[Tapir]

On Wednesday, March 26, 2014 8:04:41 AM UTC+8, Wesley C (Google) wrote:

@pdknsk: Correct, this feature has been announced publicly now so thanks for sending a link to the developer docs. To see a summary of all the exciting announcements we made today (2014 Mar 25), we suggest you check the blogpost (http://googlecloudplatform.blogspot.com/2014/03/google-cloud-platform-live-blending-iaas-and-paas-moores-law-for-the-cloud.html) and go to http://cloud.google.com for more information.

@tapir: Last year at Google I/O 2013, we launched (http://googledevelopers.blogspot.com/2013/05/get-started-with-google-cloud-datastore.html) the Google Cloud Datastore, which is exactly what you're asking for... access to the Google App Engine datastore without the need for App Engine, accessible from Compute Engine or from anywhere else with an API. The docs can be found at http://developers.google.com/datastore.

Thanks for the information!

[Goran Popovic]

Any news on when will vm backends be available for all users?

[Trung Pham]

Do you know if I can access Google datastore using the appengine SDK rather than the JSON api from the VM backend? This will certainly make things much easier for me.

Appengine SDK https://developers.google.com/appengine/docs/go/datastore/reference

[Rafael]

Also in my case. The only reason why we stick with appengine and are not on GCE is because of the datastore lock-in. 

Java being Java, I don't know how difficult it would be to abstract the JSON api into the JDO and Datastore java interfaces? 

On Wed, Apr 9, 2014 at 12:48 AM, Trung Pham <tr...@phamcom.com> wrote:

Do you know if I can access Google datastore using the appengine SDK rather than the JSON api from the VM backend? This will certainly make things much easier for me.

Appengine SDK https://developers.google.com/appengine/docs/go/datastore/reference

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.

For more options, visit https://groups.google.com/d/optout.

[Vinny P]

On Wed, Apr 9, 2014 at 2:48 AM, Trung Pham <tr...@phamcom.com> wrote:

Do you know if I can access Google datastore using the appengine SDK rather than the JSON api from the VM backend? This will certainly make things much easier for me.

Appengine SDK https://developers.google.com/appengine/docs/go/datastore/reference

Assuming you're using the Go runtime (since you linked to the Go documentation): the answer is yes. There's a recent post on google-appengine-go announcing the availability of Managed VMs for Go: https://groups.google.com/forum/#!topic/google-appengine-go/nFpBkdoNHWw  . You can use the standard App Engine services (datastore, task queues, etc) from managed VMs.

 

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

[Trung Pham]

This is great!

[Chris Ramsdale]

yes, you can use JDO, JPA, Objectify, etc. when running on a Managed VM (previously referred to as "VM Backend").  let me know if you need more clarification.

-- Chris