Google App Engine Roadmap - Now Published
31 views
Skip to first unread message
Marzia Niccolai
Oct 23, 2008, 8:45:30 PM10/23/08
to google-a...@googlegroups.com
Hi,
Many of you have expressed interest in learning about what's coming next for Google App Engine, and although we've often talked about features we plan on supporting, we've decided to publish a roadmap along with our documentation. The roadmap can be found here: http://code.google.com/appengine/docs/roadmap.html
It should be noted that while we're working hard on these features, development schedules do slip at times, and the dates below may change. We'll do our best to update this roadmap as our engineers continue development.
Without further ado, here it is:
10/08 - 3/09
* Service for storing and serving large files
* Datastore import and export utility for large datasets
* Billing: developers can pay for more resource usage
* Support for a new runtime language
* Uptime monitoring site
And, for reference, here is the release timeline since April:
* 10/16/08 - HTTPS support for *.appspot.com
* 10/14/20 - Logs for Admin Console usage, and regex filtering for application logs
* 9/18/08 - CPU usage details added to logs and dashboard, zipimport, zipserve, memcache viewer
* 8/22/08 - Multi-entity group batch writes and deletes
* 7/24/08 - Logs export capability, more apps for everyone!
* 5/28/08 - Memcache and Image Manipulation API, Open Signups
* 5/15/08 - App Engine Launcher for Mac
* 4/8/08 - App Engine Limited Preview Release launches!
-Marzia
Many of you have expressed interest in learning about what's coming next for Google App Engine, and although we've often talked about features we plan on supporting, we've decided to publish a roadmap along with our documentation. The roadmap can be found here: http://code.google.com/appengine/docs/roadmap.html
It should be noted that while we're working hard on these features, development schedules do slip at times, and the dates below may change. We'll do our best to update this roadmap as our engineers continue development.
Without further ado, here it is:
10/08 - 3/09
* Service for storing and serving large files
* Datastore import and export utility for large datasets
* Billing: developers can pay for more resource usage
* Support for a new runtime language
* Uptime monitoring site
And, for reference, here is the release timeline since April:
* 10/16/08 - HTTPS support for *.appspot.com
* 10/14/20 - Logs for Admin Console usage, and regex filtering for application logs
* 9/18/08 - CPU usage details added to logs and dashboard, zipimport, zipserve, memcache viewer
* 8/22/08 - Multi-entity group batch writes and deletes
* 7/24/08 - Logs export capability, more apps for everyone!
* 5/28/08 - Memcache and Image Manipulation API, Open Signups
* 5/15/08 - App Engine Launcher for Mac
* 4/8/08 - App Engine Limited Preview Release launches!
-Marzia
cz
Oct 23, 2008, 9:06:33 PM10/23/08
to Google App Engine
This is fantastic, thanks Marzia!
There also seems to be a fair amount of clamor for beefing up the
Image API, any chance that might be in the pipeline?
Also, I would like to suggest that the documentation be fattened up,
especially concerning the inner workings of datastore/BigTable and
related best practices. The google-appengine forum and others have a
lot of traffic concerning how to use the datastore efficiently, such
as discussions about normalizing/denormalizing model schemas etc. and
it would be great if there was some authoritative information about
this subject.
Anyways, it's altogether an awesome platform so thank you Googly types.
There also seems to be a fair amount of clamor for beefing up the
Image API, any chance that might be in the pipeline?
Also, I would like to suggest that the documentation be fattened up,
especially concerning the inner workings of datastore/BigTable and
related best practices. The google-appengine forum and others have a
lot of traffic concerning how to use the datastore efficiently, such
as discussions about normalizing/denormalizing model schemas etc. and
it would be great if there was some authoritative information about
this subject.
Anyways, it's altogether an awesome platform so thank you Googly types.
Greg
Oct 23, 2008, 9:50:43 PM10/23/08
to Google App Engine
Thanks for the feedback - good to know you are listening to our
concerns.
The big one for me is billing, which presumably will remove all the
quota issues. I have an application just about ready to go live, and I
don't anticipate huge resource use, but I'm hesitant about launching
until I can be sure customers aren't confronted with nasty quota
warnings.
Also I don't see any mention of a production release. A few hints have
suggested this will be before the end of 2008,but it would be good to
have this confirmed or denied. It would seem logical to move out of
preview when billing starts - is this likely?
Cheers!
Greg.
concerns.
The big one for me is billing, which presumably will remove all the
quota issues. I have an application just about ready to go live, and I
don't anticipate huge resource use, but I'm hesitant about launching
until I can be sure customers aren't confronted with nasty quota
warnings.
Also I don't see any mention of a production release. A few hints have
suggested this will be before the end of 2008,but it would be good to
have this confirmed or denied. It would seem logical to move out of
preview when billing starts - is this likely?
Cheers!
Greg.
Bill
Oct 23, 2008, 10:11:21 PM10/23/08
to Google App Engine
> * Support for a new runtime language
Would you be able to say how different languages will be partitioned?
Could you access a single datastore with more than one language or
have different languages running on the same VM?
Peter Recore
Oct 23, 2008, 10:38:59 PM10/23/08
to Google App Engine
"Support for a new runtime language"
What an evil way to tease all those python-a-phobes out there! Should
we start a pool on what the mystery language is? I've got $10 that
says java.
oh yeah, and Bill's question is a great one.
-peter
Jeffrey Rosen
Oct 23, 2008, 11:01:23 PM10/23/08
to Google App Engine
Thanks for clarifying this! I am especially relieved to see the
datastore import/export and the billing option announced. I am also
relieved to see recent advances like the HTTPS release which now lets
us add things like Google Checkout integration. As someone who runs
their entire business (http://www.wolfire.com) on Google App Engine,
here are a few things that I would like to see, for what it's worth:
- The latency for using your own domain. The appspot url is
distributed around the world and extremely fast. We're talking 2 -
8ms in the bay area. However, once you put your app on your own
domain, that latency goes to ~100 ms to 200ms in the bay area. Still
"decent", but slower than what a typical web host would provide. I
would be interested to hear what Google is planning to do about this.
Here is the ticket: http://code.google.com/p/googleappengine/issues/detail?id=531
For what it's worth, the reason I signed up for this service in the
first place is because I was so impressed with the CDN aspect of the
service and the low latency around the world of my pages. I don't
want to tell you guys about your own business, but IMHO, this CDN
aspect of GAE is one of the few things that makes Google unique from
other web hosting providers and Google is one of the few companies out
there that has the necessary infrastructure to pull it off. It's a
shame that GAE isn't reaching its full potential in this regard.
From what I hear, it's internal bureaucracy with the Google Apps team
handling the DNS routing and the GAE handling the other aspect of the
service, but whatever it is, I am sure you guys can solve it.
It's just really frustrating because when I was initially testing it
in jeffr.appspot.com, the site is so fast, it is literally
instantaneous. Like you click the link, and it is already there.
Even people who aren't tech savvy would comment independently on how
it was the fastest site they had ever seen, but now wolfire.com is
actually slower than it was on our original dedicated server.
- Support for naked domains. It's annoying that I have to redirect
people who go to wolfire.com to www.wolfire.com. If anything, I'd
rather do it the other way around. This also adds to the latency,
ruining our chance for an instantaneous load like on appspot.
http://code.google.com/p/googleappengine/issues/detail?id=777
- Some fixes for small misc bugs and "features". For example, it is
impossible to set a display name on emails. This is a simple best
practice to make your email look like it was from "Wolfire Games"
rather than jef...@gmail.com.
http://code.google.com/p/googleappengine/issues/detail?id=677
Another example is how GAE doesn't let you set cookies on a 301
redirect.
http://code.google.com/p/googleappengine/issues/detail?id=784
I am sure these are tiny bugs that could be corrected in less than a
day of an engineer's work yet they are actually quite important to GAE
users (despite the small number of stars). Basically, I would like a
little more transparency on what the GAE is doing. It would be cool
to maybe allocate a week to just fix all of the low hanging fruit bugs
like these ones rather than spending months working on adding Java
support or HTTPS support exclusively. I can't really comment on what
Google is doing, because I simply don't know, but that is what it
feels like to me.
A few other things to throw out there; a task queue so we can get rid
of the third party server regular http ping hack and full text search.
Sorry if this post sounds negative. I love GAE and don't regret
choosing it as my platform for a minute, but that's my feedback as
someone who works with it every day. :)
On Oct 23, 5:45 pm, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> Many of you have expressed interest in learning about what's coming next for
> Google App Engine, and although we've often talked about features we plan on
> supporting, we've decided to publish a
> roadmap<http://code.google.com/appengine/docs/roadmap.html>along with
datastore import/export and the billing option announced. I am also
relieved to see recent advances like the HTTPS release which now lets
us add things like Google Checkout integration. As someone who runs
their entire business (http://www.wolfire.com) on Google App Engine,
here are a few things that I would like to see, for what it's worth:
- The latency for using your own domain. The appspot url is
distributed around the world and extremely fast. We're talking 2 -
8ms in the bay area. However, once you put your app on your own
domain, that latency goes to ~100 ms to 200ms in the bay area. Still
"decent", but slower than what a typical web host would provide. I
would be interested to hear what Google is planning to do about this.
Here is the ticket: http://code.google.com/p/googleappengine/issues/detail?id=531
For what it's worth, the reason I signed up for this service in the
first place is because I was so impressed with the CDN aspect of the
service and the low latency around the world of my pages. I don't
want to tell you guys about your own business, but IMHO, this CDN
aspect of GAE is one of the few things that makes Google unique from
other web hosting providers and Google is one of the few companies out
there that has the necessary infrastructure to pull it off. It's a
shame that GAE isn't reaching its full potential in this regard.
From what I hear, it's internal bureaucracy with the Google Apps team
handling the DNS routing and the GAE handling the other aspect of the
service, but whatever it is, I am sure you guys can solve it.
It's just really frustrating because when I was initially testing it
in jeffr.appspot.com, the site is so fast, it is literally
instantaneous. Like you click the link, and it is already there.
Even people who aren't tech savvy would comment independently on how
it was the fastest site they had ever seen, but now wolfire.com is
actually slower than it was on our original dedicated server.
- Support for naked domains. It's annoying that I have to redirect
people who go to wolfire.com to www.wolfire.com. If anything, I'd
rather do it the other way around. This also adds to the latency,
ruining our chance for an instantaneous load like on appspot.
http://code.google.com/p/googleappengine/issues/detail?id=777
- Some fixes for small misc bugs and "features". For example, it is
impossible to set a display name on emails. This is a simple best
practice to make your email look like it was from "Wolfire Games"
rather than jef...@gmail.com.
http://code.google.com/p/googleappengine/issues/detail?id=677
Another example is how GAE doesn't let you set cookies on a 301
redirect.
http://code.google.com/p/googleappengine/issues/detail?id=784
I am sure these are tiny bugs that could be corrected in less than a
day of an engineer's work yet they are actually quite important to GAE
users (despite the small number of stars). Basically, I would like a
little more transparency on what the GAE is doing. It would be cool
to maybe allocate a week to just fix all of the low hanging fruit bugs
like these ones rather than spending months working on adding Java
support or HTTPS support exclusively. I can't really comment on what
Google is doing, because I simply don't know, but that is what it
feels like to me.
A few other things to throw out there; a task queue so we can get rid
of the third party server regular http ping hack and full text search.
Sorry if this post sounds negative. I love GAE and don't regret
choosing it as my platform for a minute, but that's my feedback as
someone who works with it every day. :)
On Oct 23, 5:45 pm, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> Many of you have expressed interest in learning about what's coming next for
> Google App Engine, and although we've often talked about features we plan on
> supporting, we've decided to publish a
Arun Shanker Prasad
Oct 24, 2008, 1:06:18 AM10/24/08
to Google App Engine
Great to finally get official about the Roadmap....
> people who go to wolfire.com towww.wolfire.com. If anything, I'd
> rather do it the other way around. This also adds to the latency,
> ruining our chance for an instantaneous load like on appspot.http://code.google.com/p/googleappengine/issues/detail?id=777
>
> - Some fixes for small misc bugs and "features". For example, it is
> impossible to set a display name on emails. This is a simple best
> practice to make your email look like it was from "Wolfire Games"
> rather than jeff...@gmail.com.http://code.google.com/p/googleappengine/issues/detail?id=677
> - Some fixes for small misc bugs and "features". For example, it is
> impossible to set a display name on emails. This is a simple best
> practice to make your email look like it was from "Wolfire Games"
>
> Another example is how GAE doesn't let you set cookies on a 301
> redirect.http://code.google.com/p/googleappengine/issues/detail?id=784
> Another example is how GAE doesn't let you set cookies on a 301
nslindtner
Oct 24, 2008, 2:03:17 AM10/24/08
to Google App Engine
Nice with some roadmap information, and also neccesary as i see it.
But to my big disappointment I do not se long-running processes .-(.
I'm very well aware that it raises a lot of quistions, but come-on.
Using webcron (and keeping every request på 8 sec) seems like a lot of
unneccesary enginering - all together not having to deal with this
kind of enginering is why app engine is so appealing.
Keep up the great work ... and take a look around at the
competition ... this is the time for you guys to get a leap to MS and
yaaho
On 24 Okt., 07:06, Arun Shanker Prasad <ArunShankerPra...@gmail.com>
wrote:
But to my big disappointment I do not se long-running processes .-(.
I'm very well aware that it raises a lot of quistions, but come-on.
Using webcron (and keeping every request på 8 sec) seems like a lot of
unneccesary enginering - all together not having to deal with this
kind of enginering is why app engine is so appealing.
Keep up the great work ... and take a look around at the
competition ... this is the time for you guys to get a leap to MS and
yaaho
On 24 Okt., 07:06, Arun Shanker Prasad <ArunShankerPra...@gmail.com>
wrote:
Jon McAlister
Oct 24, 2008, 4:45:30 AM10/24/08
to Google App Engine
Hi Jeffrey, I appreciate your feedback. You definitely have good
insight into the issues and the reasons why they are still issues :-).
We recently did such a fixit of low hanging user bugs and these fixes
will be pushed soon. I don't remember if we got everything that you
mentioned in your email, as we focused on the most highly requested
bug fixes.
We are also still interested in offline processing and a task queue
API. In fact, there are tons of things that we are investigating and
implementing that aren't reflected on the roadmap. We are
intentionally trying to under-promise and over-deliver, and having a
roadmap at all is a fairly novel and experimental thing at Google (as
is the uptime monitoring site we're working on).
I hope we can get your issues resolved quickly. It's hard to balance
new feature development, bug fixing, and service maintenance, but we
sincerely do not want to fall behind on bug fixing.
Jon
> people who go to wolfire.com towww.wolfire.com. If anything, I'd
insight into the issues and the reasons why they are still issues :-).
We recently did such a fixit of low hanging user bugs and these fixes
will be pushed soon. I don't remember if we got everything that you
mentioned in your email, as we focused on the most highly requested
bug fixes.
We are also still interested in offline processing and a task queue
API. In fact, there are tons of things that we are investigating and
implementing that aren't reflected on the roadmap. We are
intentionally trying to under-promise and over-deliver, and having a
roadmap at all is a fairly novel and experimental thing at Google (as
is the uptime monitoring site we're working on).
I hope we can get your issues resolved quickly. It's hard to balance
new feature development, bug fixing, and service maintenance, but we
sincerely do not want to fall behind on bug fixing.
Jon
> rather do it the other way around. This also adds to the latency,
> ruining our chance for an instantaneous load like on appspot.http://code.google.com/p/googleappengine/issues/detail?id=777
>
> - Some fixes for small misc bugs and "features". For example, it is
> impossible to set a display name on emails. This is a simple best
> practice to make your email look like it was from "Wolfire Games"
> rather than jeff...@gmail.com.http://code.google.com/p/googleappengine/issues/detail?id=677
> - Some fixes for small misc bugs and "features". For example, it is
> impossible to set a display name on emails. This is a simple best
> practice to make your email look like it was from "Wolfire Games"
>
> Another example is how GAE doesn't let you set cookies on a 301
> redirect.http://code.google.com/p/googleappengine/issues/detail?id=784
> Another example is how GAE doesn't let you set cookies on a 301
Nash
Oct 24, 2008, 10:25:16 AM10/24/08
to Google App Engine
This is great! Thanks Marzia et al! It's a great step towards giving
us visibility to what we should expect and what we should be planning
for.
On Oct 24, 5:45 am, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> Many of you have expressed interest in learning about what's coming next for
> Google App Engine, and although we've often talked about features we plan on
> supporting, we've decided to publish a
> roadmap<http://code.google.com/appengine/docs/roadmap.html>along with
us visibility to what we should expect and what we should be planning
for.
On Oct 24, 5:45 am, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> Many of you have expressed interest in learning about what's coming next for
> Google App Engine, and although we've often talked about features we plan on
> supporting, we've decided to publish a
Marzia Niccolai
Oct 24, 2008, 2:20:17 PM10/24/08
to google-a...@googlegroups.com
Hi,
First to further what Jon was saying, the roadmap has to do with our feature development plans, and not with issue fixes. In addition to building new features, the team will also continue fixing issues with the system. These include those issues listed in the tracker, as well as an overall dedication to improving system performance.
To address the comments regarding the documentation, improvements to the documentation are outside of the scope of this roadmap, but we definitely are committed to improving that as well.
I'd also like this opportunity to solicit user contributed articles for our articles section:
http://code.google.com/appengine/articles/
If you think you have a good idea for an article, and are willing to license it under the Creative Commons license, please contact me and I can work with you to get it published.
Lastly, concerning Bill's comment, I'm not entirely sure what you have in mind when you ask if they will be running on the same 'VM', immediate plans for new language support would most likely follow the same general theme as python support. Meaning, you would upload your application to App Engine, and the system manages all of serving infrastructure.
-Marzia
First to further what Jon was saying, the roadmap has to do with our feature development plans, and not with issue fixes. In addition to building new features, the team will also continue fixing issues with the system. These include those issues listed in the tracker, as well as an overall dedication to improving system performance.
To address the comments regarding the documentation, improvements to the documentation are outside of the scope of this roadmap, but we definitely are committed to improving that as well.
I'd also like this opportunity to solicit user contributed articles for our articles section:
http://code.google.com/appengine/articles/
If you think you have a good idea for an article, and are willing to license it under the Creative Commons license, please contact me and I can work with you to get it published.
Lastly, concerning Bill's comment, I'm not entirely sure what you have in mind when you ask if they will be running on the same 'VM', immediate plans for new language support would most likely follow the same general theme as python support. Meaning, you would upload your application to App Engine, and the system manages all of serving infrastructure.
-Marzia
Bill
Oct 24, 2008, 3:27:45 PM10/24/08
to Google App Engine
> Lastly, concerning Bill's comment, I'm not entirely sure what you have in
> mind when you ask if they will be running on the same 'VM', immediate plans
> for new language support would most likely follow the same general theme as
> python support. Meaning, you would upload your application to App Engine,
> and the system manages all of serving infrastructure.
Not knowing how apps really run in the cloud, I wondered if current
> mind when you ask if they will be running on the same 'VM', immediate plans
> for new language support would most likely follow the same general theme as
> python support. Meaning, you would upload your application to App Engine,
> and the system manages all of serving infrastructure.
python (and possibly future languages) got compiled to some App Engine
VM which might allow cross-language calls (like using JVM for JRuby or
the .NET CLR). It sounds like this is not the case but can different
language files coexist within a single app and access the same
datastore?
Marzia Niccolai
Oct 24, 2008, 5:59:10 PM10/24/08
to google-a...@googlegroups.com
Ah, I understand the question now. For the Python runtime, the App Engine frontend actually sends requests for your app to one or more Python Interpreters for your app.
The I/O talk 'Building a Production Quality Application on Google App Engine' actually deals with this topic when talking about how to run loadtests with your app. (Around slide 49 of the talk.)
The docs also mention it when talking about the Python runtime (http://code.google.com/appengine/docs/python/):
'The App Engine Python runtime environment includes a specialized version of the Python interpreter, the standard Python library, libraries and APIs for App Engine, and a standard interface to the web server layer.'
Concerning your question, at this time I don't have an answer as to how different runtimes may or may not be able to interact using the same app id (rest assured it will be clarified prior to any launch).
-Marzia
The I/O talk 'Building a Production Quality Application on Google App Engine' actually deals with this topic when talking about how to run loadtests with your app. (Around slide 49 of the talk.)
The docs also mention it when talking about the Python runtime (http://code.google.com/appengine/docs/python/):
'The App Engine Python runtime environment includes a specialized version of the Python interpreter, the standard Python library, libraries and APIs for App Engine, and a standard interface to the web server layer.'
Concerning your question, at this time I don't have an answer as to how different runtimes may or may not be able to interact using the same app id (rest assured it will be clarified prior to any launch).
-Marzia
ryan
Oct 24, 2008, 9:39:37 PM10/24/08
to Google App Engine
the short answer is, at least in the short to medium term, each app
will be limited to a single language. multi-language apps, sharing
datastores across apps, and inter-app RPCs are definitely interesting
ideas, and we've had fun discussing them internally. they're not on
the roadmap, though, and we're not working on them right now.
will be limited to a single language. multi-language apps, sharing
datastores across apps, and inter-app RPCs are definitely interesting
ideas, and we've had fun discussing them internally. they're not on
the roadmap, though, and we're not working on them right now.
cm_gui
Nov 2, 2008, 5:46:01 PM11/2/08
to Google App Engine
Dear Marzia
Can you say a bit more about the Service for storing and serving large
files?
Actually, I want to write a html form with file upload. After the
form is submitted,
an email will be sent out containing the form data and a link to
download the
uploaded files.
Will the Service for storing and serving large files provide a file
system
to store the uploaded files from forms?
Thank you.
Gcm
On Oct 23, 4:45 pm, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> Many of you have expressed interest in learning about what's coming next for
> Google App Engine, and although we've often talked about features we plan on
> supporting, we've decided to publish a
> roadmap<http://code.google.com/appengine/docs/roadmap.html>along with
Can you say a bit more about the Service for storing and serving large
files?
Actually, I want to write a html form with file upload. After the
form is submitted,
an email will be sent out containing the form data and a link to
download the
uploaded files.
Will the Service for storing and serving large files provide a file
system
to store the uploaded files from forms?
Thank you.
Gcm
On Oct 23, 4:45 pm, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> Many of you have expressed interest in learning about what's coming next for
> Google App Engine, and although we've often talked about features we plan on
> supporting, we've decided to publish a
johnP
Nov 3, 2008, 5:14:58 PM11/3/08
to Google App Engine
I still don't understand one aspect of Google's approach - regarding
CPU limits. For example, consider my application, a personal
organizer. Each page view needs to tie together disparate entities;
there are lots of writes; etc. The final result of the application is
the creation of reports (using reportlab). Here, I need to make some
queries, perform some calculations, etc.
In setting up the organizer, there are several tasks that are somewhat
large. I get CPU warnings for being, say, 3-5 times over the average
limit. Some common pageviews are right around the limit (1.0, 1.1
times over the limit).
Finally, when generating reports, I might get some warnings that my
app is 4-5 times over the limit.
The question is: is an application of my sort suitable for Appengine?
Obviously, I'll optimize, and hope to get a majority of my common
pageviews into limits. But it is inevitable that a significant
percentage of requests will exceed CPU quotas.
I am willing to pay money for exceeding quotas...
There is still a significant amount of development ahead. If it
appears that my task is a non-starter for Appengine, I'd prefer to
switch to pure django earlier rather than later.
Can anyone provide some enlightenment for my dilemma? Thanks.
CPU limits. For example, consider my application, a personal
organizer. Each page view needs to tie together disparate entities;
there are lots of writes; etc. The final result of the application is
the creation of reports (using reportlab). Here, I need to make some
queries, perform some calculations, etc.
In setting up the organizer, there are several tasks that are somewhat
large. I get CPU warnings for being, say, 3-5 times over the average
limit. Some common pageviews are right around the limit (1.0, 1.1
times over the limit).
Finally, when generating reports, I might get some warnings that my
app is 4-5 times over the limit.
The question is: is an application of my sort suitable for Appengine?
Obviously, I'll optimize, and hope to get a majority of my common
pageviews into limits. But it is inevitable that a significant
percentage of requests will exceed CPU quotas.
I am willing to pay money for exceeding quotas...
There is still a significant amount of development ahead. If it
appears that my task is a non-starter for Appengine, I'd prefer to
switch to pure django earlier rather than later.
Can anyone provide some enlightenment for my dilemma? Thanks.
Sudhir Jonathan
Nov 4, 2008, 9:19:50 AM11/4/08
to Google App Engine
Yeah, John has a point... when the billing system is implemented, will
you guys relax the limits on CPU optimization per request? Avoiding
infinite loops or deadlocks is fine, but there are resource hungry
calculations we have to do... I'm gonna need to calculate the centers
and distributions of tens of thousands of map points, and its a no
starter / course in advanced mathematics and algorithms if I can't
just use brute force.
Sudhir
you guys relax the limits on CPU optimization per request? Avoiding
infinite loops or deadlocks is fine, but there are resource hungry
calculations we have to do... I'm gonna need to calculate the centers
and distributions of tens of thousands of map points, and its a no
starter / course in advanced mathematics and algorithms if I can't
just use brute force.
Sudhir
Rodrigo Moraes
Nov 4, 2008, 8:08:30 AM11/4/08
to google-a...@googlegroups.com
On Mon, Nov 3, 2008 at 8:14 PM, johnP wrote:
> There is still a significant amount of development ahead. If it
> appears that my task is a non-starter for Appengine, I'd prefer to
> switch to pure django earlier rather than later.
>
> Can anyone provide some enlightenment for my dilemma? Thanks.
> There is still a significant amount of development ahead. If it
> appears that my task is a non-starter for Appengine, I'd prefer to
> switch to pure django earlier rather than later.
>
> Can anyone provide some enlightenment for my dilemma? Thanks.
I think this is a common concern. I don't want my app offline because
I'm exceeding limits. Instead, I'd like to pay to be allowed to exceed
limits - and it is on Google plans to give us this option (from the
roadmap: "Billing: developers can pay for more resource usage"). I
have great expectations and I'm waiting for it.
-- rodrigo
johnP
Nov 4, 2008, 11:46:15 AM11/4/08
to Google App Engine
On Nov 4, 5:08 am, "Rodrigo Moraes" <rodrigo.mor...@gmail.com> wrote:
> On Mon, Nov 3, 2008 at 8:14 PM, johnP wrote:
> roadmap: "Billing: developers can pay for more resource usage"). I
>
It's not clear what "paying for more resource usage" means. Yes, you
can buy more traffic, storage, etc. It is *not* clear that apps
exceeding quotas for CPU-intensive tasks will not be taken offline.
I hope the Google folks can give us a clear answer here. We need to
to understand if the platform is suitable for our needs. I am willing
to deal with BigTable datastore limitations if my application is
ultimately viable for the platform. If not, I'd prefer to develop
with all the capabilities of SQL at my disposal.
Marzia Niccolai
Nov 4, 2008, 7:48:34 PM11/4/08
to google-a...@googlegroups.com
Hi,
Billing will allow you to scale your application to your heart's content. So, in general, most quotas will no longer exist when billing is enabled. However, the per-request High CPU limits in App Engine are in place to protect the stability and responsiveness of the cluster as a whole. We know that this solution is not great, and we are working hard on fixing this, so that your apps don't hit this stumbling block. To be clear, we consider limiting how much CPU you can use in a single request to be a bug, and we're working hard on fixing this issue for everyone.
So, the short answer is: the per-request CPU usage limits is *very* much on our radar, and while we may not have a solution in time for the release of Billing, we're working hard on it!
As for the question of uploading and storing large files - the example given, uploading large files in a form, is definitely something we are planning on supporting.
-Marzia
Billing will allow you to scale your application to your heart's content. So, in general, most quotas will no longer exist when billing is enabled. However, the per-request High CPU limits in App Engine are in place to protect the stability and responsiveness of the cluster as a whole. We know that this solution is not great, and we are working hard on fixing this, so that your apps don't hit this stumbling block. To be clear, we consider limiting how much CPU you can use in a single request to be a bug, and we're working hard on fixing this issue for everyone.
So, the short answer is: the per-request CPU usage limits is *very* much on our radar, and while we may not have a solution in time for the release of Billing, we're working hard on it!
As for the question of uploading and storing large files - the example given, uploading large files in a form, is definitely something we are planning on supporting.
-Marzia
johnP
Nov 4, 2008, 8:07:44 PM11/4/08
to Google App Engine
Thank you for the response. It makes me feel better about investing
time into creating a significant app on the platform. Like I
mentioned, the core concept of keeping each pageview quick, is one
that I wholeheartedly support. I think that a fast site is probably
the most important usability factor there is. But in my case, the end
result is reports. And having a few extra seconds to generate and
calculate a usable report will be the difference between a quality
outcome, and a useless one.
I wish you the best of luck in addressing this bug as quickly as
possible.
johnP
On Nov 4, 4:48 pm, Marzia Niccolai <ma...@google.com> wrote:
> Hi,
>
> Billing will allow you to scale your application to your heart's content.
> So, in general, most quotas will no longer exist when billing is enabled.
> However, the per-request High CPU limits in App Engine are in place to
> protect the stability and responsiveness of the cluster as a whole. We know
> that this solution is not great, and we are working hard on fixing this, so
> that your apps don't hit this stumbling block. To be clear, we consider
> limiting how much CPU you can use in a single request to be a bug, and we're
> working hard on fixing this issue for everyone.
>
> So, the short answer is: the per-request CPU usage limits is *very* much on
> our radar, and while we may not have a solution in time for the release of
> Billing, we're working hard on it!
>
> As for the question of uploading and storing large files - the example
> given, uploading large files in a form, is definitely something we are
> planning on supporting.
>
> -Marzia
>
time into creating a significant app on the platform. Like I
mentioned, the core concept of keeping each pageview quick, is one
that I wholeheartedly support. I think that a fast site is probably
the most important usability factor there is. But in my case, the end
result is reports. And having a few extra seconds to generate and
calculate a usable report will be the difference between a quality
outcome, and a useless one.
I wish you the best of luck in addressing this bug as quickly as
possible.
johnP
On Nov 4, 4:48 pm, Marzia Niccolai <ma...@google.com> wrote:
> Hi,
>
> Billing will allow you to scale your application to your heart's content.
> So, in general, most quotas will no longer exist when billing is enabled.
> However, the per-request High CPU limits in App Engine are in place to
> protect the stability and responsiveness of the cluster as a whole. We know
> that this solution is not great, and we are working hard on fixing this, so
> that your apps don't hit this stumbling block. To be clear, we consider
> limiting how much CPU you can use in a single request to be a bug, and we're
> working hard on fixing this issue for everyone.
>
> So, the short answer is: the per-request CPU usage limits is *very* much on
> our radar, and while we may not have a solution in time for the release of
> Billing, we're working hard on it!
>
> As for the question of uploading and storing large files - the example
> given, uploading large files in a form, is definitely something we are
> planning on supporting.
>
> -Marzia
>
Aral Balkan
Nov 5, 2008, 10:54:59 AM11/5/08
to Google App Engine
The roadmap is a great step forward, thank you.
I am, however, disappointed at seeing "Support for a new runtime
language" on the roadmap for the immediate feature.
Quite bluntly, I would have expected better triage.
There are showstopping issues in Google App Engine at the moment.
Applications that require reporting, or any sort of maintenance
features/long-running processes have to implement quite ridiculous
client-side hacks. My app has webbasedcron hitting a URL every minute
to run its email queue. If I want to email everyone, I run my admin
routine which uses Ajax calls to iterate over the several thousand
members. When I send an email from my app I have no idea what happens
to it (and we had such a hard time because said email was ending up in
users' spam folders.
Google App Engine is missing an entire mode of operation that is
required for web apps. It handles lots of tiny request/response
transactions well but you also need to be able to run reporting and
maintenance. Long-running processes that *do not need to scale* but
are nonetheless essential.
I don't care that I will soon be able to create such hacks in another
language. I'd like those issues addressed first before time and effort
is expanded on a feature that is purely one of preference. As I
mentioned in a recent post, adding support for other languages to
Google App Engine today is like sewing a new set of drapes for a house
that doesn't have any walls yet. I'd rather have the walls instead of
a "nice to have" vanity feature.
Thanks,
Aral
I am, however, disappointed at seeing "Support for a new runtime
language" on the roadmap for the immediate feature.
Quite bluntly, I would have expected better triage.
There are showstopping issues in Google App Engine at the moment.
Applications that require reporting, or any sort of maintenance
features/long-running processes have to implement quite ridiculous
client-side hacks. My app has webbasedcron hitting a URL every minute
to run its email queue. If I want to email everyone, I run my admin
routine which uses Ajax calls to iterate over the several thousand
members. When I send an email from my app I have no idea what happens
to it (and we had such a hard time because said email was ending up in
users' spam folders.
Google App Engine is missing an entire mode of operation that is
required for web apps. It handles lots of tiny request/response
transactions well but you also need to be able to run reporting and
maintenance. Long-running processes that *do not need to scale* but
are nonetheless essential.
I don't care that I will soon be able to create such hacks in another
language. I'd like those issues addressed first before time and effort
is expanded on a feature that is purely one of preference. As I
mentioned in a recent post, adding support for other languages to
Google App Engine today is like sewing a new set of drapes for a house
that doesn't have any walls yet. I'd rather have the walls instead of
a "nice to have" vanity feature.
Thanks,
Aral
Aral Balkan
Nov 5, 2008, 12:05:43 PM11/5/08
to Google App Engine
To illustrate my point further with an analogy:
Google App Engine is here right now:
http://icanhaz.com/rightscaleweb
It needs to be here:
http://icanhaz.com/rightscalepremium
Aral
On Nov 5, 3:54 pm, Aral Balkan <aralbal...@gmail.com> wrote:
> The roadmap is a great step forward, thank you.
>
> I am, however, disappointed at seeing "Support for a new runtime
> language" on the roadmap for the immediate feature.
>
> Quite bluntly, I would have expected better triage.
<snip>
Google App Engine is here right now:
http://icanhaz.com/rightscaleweb
It needs to be here:
http://icanhaz.com/rightscalepremium
Aral
On Nov 5, 3:54 pm, Aral Balkan <aralbal...@gmail.com> wrote:
> The roadmap is a great step forward, thank you.
>
> I am, however, disappointed at seeing "Support for a new runtime
> language" on the roadmap for the immediate feature.
>
> Quite bluntly, I would have expected better triage.
Edoardo Marcora
Nov 6, 2008, 12:04:46 PM11/6/08
to Google App Engine
Ciao Marzia -
I forgot to ask you this question yesterday during the "Chat with the
GAE team" session (which, by the way, I think is a great idea).
Will the "service for storing and serving large files" be something
like S3, totally separate from the datastore, or would it be in the
form of a new property type (e.g., FileProperty) or (preferred by me)
a BlobProperty without the current 1MB size limit???
Thanx,
Dado
On Oct 23, 4:45 pm, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> Many of you have expressed interest in learning about what's coming next for
> Google App Engine, and although we've often talked about features we plan on
> supporting, we've decided to publish a
> roadmap<http://code.google.com/appengine/docs/roadmap.html>along with
I forgot to ask you this question yesterday during the "Chat with the
GAE team" session (which, by the way, I think is a great idea).
Will the "service for storing and serving large files" be something
like S3, totally separate from the datastore, or would it be in the
form of a new property type (e.g., FileProperty) or (preferred by me)
a BlobProperty without the current 1MB size limit???
Thanx,
Dado
On Oct 23, 4:45 pm, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> Many of you have expressed interest in learning about what's coming next for
> Google App Engine, and although we've often talked about features we plan on
> supporting, we've decided to publish a
hawkett
Nov 6, 2008, 11:19:21 AM11/6/08
to Google App Engine
<black-hat>
I would like to echo Aral's post. Language wars generate noise, not
analysis. It's no surprise that the top four issues *by a mile* are
just adding language ABC. It is depressing that people are working on
that at the expense of other things.
Admittedly, I am assuming that providing a complete application
platform is the primary goal of GAE? More languages at this point
hints that the goal is to provide multiple incomplete application
platforms. Not supporting asynchronous processing is an enourmous
hole, and to not have it on the roadmap, well, it almost seems like
team members are making their own decisions about what they want to
do :).
</black-hat>
<white-hat>
I understand that as a business, you want the largest user base, and
supporting PHP or Java is going to greatly increase that - but you are
creating a rod for your own back by branching out in this way before
the platform offering is compete. The pressure to get any new
features right the first time goes up exponentially with more (and
paying) customers, and the difficulty in getting it right goes up
exponentially with more runtimes, because you have a larger, more
complex codebase.
The thing is, *I know* you are going to get round to it - I'm psychic
- but there is no way that I can tell an investor, a client, or a
business partner that I'm psychic, they just aren't that enlightened -
they go for the old-school way of making business decisions.
Client - 'Can I run a report on this thing?',
Me - 'Spirit guide says yes - soon.',
Client - 'Shut-up fool. Bob says we should be on Amazon.'
It wouldn't take much to stop that conversation from happening.
</white-hat>
<yellow-hat><red-hat>
I don't want to use another platform. BigTable is, frankly, awesome.
Its a killer feature. After using it, I don't want to contemplate the
thought of managing my own data layer again. But I have to, because
asynchronous processing isn't even on the map. :(
</red-hat></yellow-hat>
<green-hat>
Perhaps - to get asynchronous processing up the issue list you could
combine the following two issues
http://code.google.com/p/googleappengine/issues/detail?id=6
http://code.google.com/p/googleappengine/issues/detail?id=109
and just call it 'Asynchronous Processing' - at least it would get
above adding perl as a supported laguage. God forbid we couldn't
deploy a perl application to app engine.
</green-hat>
I haven't got a blue hat, because someone lost it.
I'm sure this kind of post is why you don't publish roadmaps :)
Better tack this on as well - I'd love for you to secure the python
interpreter -
http://code.google.com/p/googleappengine/issues/detail?id=671
The actual code for doing so has been submitted with this issue, and
its something you would want to do sooner rather than later so you
don't break too many existing apps that rely on it being insecure. It
looks like something you might be able to knock out fairly quickly?
I would like to echo Aral's post. Language wars generate noise, not
analysis. It's no surprise that the top four issues *by a mile* are
just adding language ABC. It is depressing that people are working on
that at the expense of other things.
Admittedly, I am assuming that providing a complete application
platform is the primary goal of GAE? More languages at this point
hints that the goal is to provide multiple incomplete application
platforms. Not supporting asynchronous processing is an enourmous
hole, and to not have it on the roadmap, well, it almost seems like
team members are making their own decisions about what they want to
do :).
</black-hat>
<white-hat>
I understand that as a business, you want the largest user base, and
supporting PHP or Java is going to greatly increase that - but you are
creating a rod for your own back by branching out in this way before
the platform offering is compete. The pressure to get any new
features right the first time goes up exponentially with more (and
paying) customers, and the difficulty in getting it right goes up
exponentially with more runtimes, because you have a larger, more
complex codebase.
The thing is, *I know* you are going to get round to it - I'm psychic
- but there is no way that I can tell an investor, a client, or a
business partner that I'm psychic, they just aren't that enlightened -
they go for the old-school way of making business decisions.
Client - 'Can I run a report on this thing?',
Me - 'Spirit guide says yes - soon.',
Client - 'Shut-up fool. Bob says we should be on Amazon.'
It wouldn't take much to stop that conversation from happening.
</white-hat>
<yellow-hat><red-hat>
I don't want to use another platform. BigTable is, frankly, awesome.
Its a killer feature. After using it, I don't want to contemplate the
thought of managing my own data layer again. But I have to, because
asynchronous processing isn't even on the map. :(
</red-hat></yellow-hat>
<green-hat>
Perhaps - to get asynchronous processing up the issue list you could
combine the following two issues
http://code.google.com/p/googleappengine/issues/detail?id=6
http://code.google.com/p/googleappengine/issues/detail?id=109
and just call it 'Asynchronous Processing' - at least it would get
above adding perl as a supported laguage. God forbid we couldn't
deploy a perl application to app engine.
</green-hat>
I haven't got a blue hat, because someone lost it.
I'm sure this kind of post is why you don't publish roadmaps :)
Better tack this on as well - I'd love for you to secure the python
interpreter -
http://code.google.com/p/googleappengine/issues/detail?id=671
The actual code for doing so has been submitted with this issue, and
its something you would want to do sooner rather than later so you
don't break too many existing apps that rely on it being insecure. It
looks like something you might be able to knock out fairly quickly?
Blessed Geek
Nov 11, 2008, 10:26:57 AM11/11/08
to Google App Engine
I tolerate javascript. I am also tolerating python.
GTK is a huge gift from google because I have become too used to to
decipher problems from cryptic java compilation logs. Familiarity
breeds contempt - contempt for the ways others do their programming.
It's hard for me to work with scripting languages because I don't want
to memorise all the pitfalls to avoid. I want the compiler to tell me
that my syntax is wrong and depend as little as possible on tracing
the output of a debugger to verify my syntax. Debugging is for
verifying logic not syntax.
Programmers who are deeply involved in PHP, python or perl find it
difficult to empathize with the attitude of the java or c++ programmer
- precision of syntax and logic. It was a huge relieve to move away
from Perl into java. GTK provides a similarly huge relief for me to
code with as little javascript as possible. We don't want to muck
around with uncertainties.
Basing app engine on a derivative of java is better than continuing
the development on python. Use of python as the initial platform was a
smart choice - for a beta test-out of the app engine concept. But as
complexity of the product grows, it's time to base it on a language
that offers more precise behaviour.
Invariably, we write quick and dirty rapid developments in perl or
python with lots of javascript to show customers a working concept. As
soon as they like it, we quickly move it into java and gtk, rather
than depend on the imprecision and lack of discipline of scripting
languages and the huge amount of mental interest needed to debug and
maintain such scripts. To free us to focus on more complex issues such
as synchronicity and security.
GTK is a huge gift from google because I have become too used to to
decipher problems from cryptic java compilation logs. Familiarity
breeds contempt - contempt for the ways others do their programming.
It's hard for me to work with scripting languages because I don't want
to memorise all the pitfalls to avoid. I want the compiler to tell me
that my syntax is wrong and depend as little as possible on tracing
the output of a debugger to verify my syntax. Debugging is for
verifying logic not syntax.
Programmers who are deeply involved in PHP, python or perl find it
difficult to empathize with the attitude of the java or c++ programmer
- precision of syntax and logic. It was a huge relieve to move away
from Perl into java. GTK provides a similarly huge relief for me to
code with as little javascript as possible. We don't want to muck
around with uncertainties.
Basing app engine on a derivative of java is better than continuing
the development on python. Use of python as the initial platform was a
smart choice - for a beta test-out of the app engine concept. But as
complexity of the product grows, it's time to base it on a language
that offers more precise behaviour.
Invariably, we write quick and dirty rapid developments in perl or
python with lots of javascript to show customers a working concept. As
soon as they like it, we quickly move it into java and gtk, rather
than depend on the imprecision and lack of discipline of scripting
languages and the huge amount of mental interest needed to debug and
maintain such scripts. To free us to focus on more complex issues such
as synchronicity and security.
hawkett
Nov 30, 2008, 10:37:21 PM11/30/08
to Google App Engine
I have to disagree with that experience - probably each to their own -
python certainly has its limitations. I only started using python for
GAE - being a J2EE guy for way too long (still). I am enjoyng the
freedom of things like closures, instant gratification (change the
server code, refresh the browser) and finding the actual delivery of
functionality is much faster. Using GWT, it is an interesting
paradigm shift having Java for client development, and the non type-
safe language on the server. The other thing I am finding is that
pushing the MVC to the client means the layer between the HTTP request
processor and the database is pretty thin - especially with BigTable/
Data API. So perhaps I don't mind Python because this architecture
means there isn't much to do - the server certainly has nothing to do
with rendering the view.
A lot of the arguments for the J2EE architecture disappear if you take
away MVC, take away EJB, take away RMI, take away hibernate & take
away clustering (which GWT+GAE does) - what's really left? Request
processing via a simple command pattern, security and data handling.
Authentication is external to the app - so you've just got
authorisation on the security front. GAE essentially delivers all the
complex parts of the J2EE architecture in the actual platform, and
makes them not my problem. So I'm not at the head of the queue to get
server side java frameworks on GAE, just so we can abstract an fairly
elegant API with a heavy one.
In the end, I might easily move to a java implementation if it came
along - there would definately be advantages - but from my
perspective, it is not as important as providing the basic building
blocks of the application platform. Not being able to do anything
aysnchronous in python *and* java seems the wrong way to go. Maybe
that's not the intention, but it would be nice to know.
I'm not saying we shouldn't have java, but that we shouldn't have it
at the expense of the actual application platform. And I'm certainly
not declaring any religous alliegance to Python.
python certainly has its limitations. I only started using python for
GAE - being a J2EE guy for way too long (still). I am enjoyng the
freedom of things like closures, instant gratification (change the
server code, refresh the browser) and finding the actual delivery of
functionality is much faster. Using GWT, it is an interesting
paradigm shift having Java for client development, and the non type-
safe language on the server. The other thing I am finding is that
pushing the MVC to the client means the layer between the HTTP request
processor and the database is pretty thin - especially with BigTable/
Data API. So perhaps I don't mind Python because this architecture
means there isn't much to do - the server certainly has nothing to do
with rendering the view.
A lot of the arguments for the J2EE architecture disappear if you take
away MVC, take away EJB, take away RMI, take away hibernate & take
away clustering (which GWT+GAE does) - what's really left? Request
processing via a simple command pattern, security and data handling.
Authentication is external to the app - so you've just got
authorisation on the security front. GAE essentially delivers all the
complex parts of the J2EE architecture in the actual platform, and
makes them not my problem. So I'm not at the head of the queue to get
server side java frameworks on GAE, just so we can abstract an fairly
elegant API with a heavy one.
In the end, I might easily move to a java implementation if it came
along - there would definately be advantages - but from my
perspective, it is not as important as providing the basic building
blocks of the application platform. Not being able to do anything
aysnchronous in python *and* java seems the wrong way to go. Maybe
that's not the intention, but it would be nice to know.
I'm not saying we shouldn't have java, but that we shouldn't have it
at the expense of the actual application platform. And I'm certainly
not declaring any religous alliegance to Python.
Reply all
Reply to author
Forward
0 new messages