Unhappy GAE Developers Petition

[Andrius]

Are you unhappy with production issues? Are you unhappy with new
pricing model? Want to have a support available or a way to contact
Google directly with the response?

Lets fill in this form collectively and get things moving rather than
being ignored:
https://docs.google.com/spreadsheet/viewform?hl=en_US&formkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

[Simon Knott]

Hi,

In my experience on the GAE forums, a lot of "unhappy" developers I see are actually the people who don't understand the GAE architecture and have poorly designed applications for the environment.  Their applications are subsequently slow and/or expensive to run, for example because they haven't denormalised their data structure or they've indexed every property under the sun, and they then lay the blame on Google.  I personally applaud the development team who provide adhoc support on these forums, as they have far more patience than I have!

I'm not saying that there couldn't be some improvements to the support channels for genuine issues with the service and I would love to have a more tiered structure, as someone has suggested in the last day, so that smaller companies could get support for less money, within a larger SLA.  It will be interesting to see who responds to your questionnaire and the complaints that are aired.

Cheers,

Simon

[Andrius A]

Yes, we all do mistakes and GAE platform is great. But this thread is not about this discussion. So if you have any outstanding issues fill in the form!

Personally I can't understand how you can be happy with price increase and production issues with python 2.7
I totally believe that for hosting just html in app engine is great but if you try using more apis such as backends, channels, task queues you will come up to lots of problems.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/WRUFpsUp6IcJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[pdknsk]

I like the rating scale.

[Simon Knott]

Maybe I'm happy because I don't use Python at all ;)

[Jeff Schnitzer]

I am not one of your unhappy developers, but I looked at your survey and it seems pretty pointless.  Let me suggest a few additions that might give it some value:

 * Java / Python / Python27?

 * Have you tried multithreading?  yes/no

 * How long have you been developing on appengine?  <3mo, 6mo, 1 yr, 2yr, more than 2yr

 * What is your monthly billing?  Free, <$10, $10-$50, $50-$100, $100-$1000, more than $1000

You can adjust the suggested numbers as you like.

You could do even better by breaking down your "how unhappy are you?" questions by component:  Datastore, Task Queue, Memcache, Channels, Email, etc.

Jeff

--

I am the 20%

[Andrius A]

Thanks for pointing Jeff, will add more fields!

[Kaan Soral]

+1

[Brian Quinlan]

Hi Andrius,

On Thu, Nov 24, 2011 at 11:40 PM, Andrius A <andr...@gmail.com> wrote:
> Yes, we all do mistakes and GAE platform is great. But this thread is not
> about this discussion. So if you have any outstanding issues fill in the
> form!
>
> Personally I can't understand how you can be happy with price increase and
> production issues with python 2.7

As an experimental release, we were expecting significant issues with
the Python 2.7 runtime (i.e. instability, missing features,
performance problems) and I've actually been pleasantly surprised at
how few have been found (thanks to everyone for helping us test!).

Are you unhappy with Python 2.7 production issues because we didn't
adequately explain that you might experience problems when using it?
Or is there some other reason?

Cheers,
Brian

[Andrius A]

Hi Brian,

I am not happy because 2.7 is still experimental but pricing is increased, and I will be even more unhappy if by 1st December with fronted price increase python 2.7 will still be experimental.

and to add to this there are more production issues I was writing but unfortunately haven't received an answer. Here is the list:

1) Broken backends in SDK: 
http://groups.google.com/group/google-appengine-python/browse_thread/... 

2) Unexplained random frontend response latency: 
http://groups.google.com/group/google-appengine/browse_thread/thread/... 

3) Memcache expiry time setting bug: 
http://groups.google.com/group/google-appengine/browse_thread/thread/... 

4) Channel API not working in backends: 
http://code.google.com/p/googleappengine/issues/detail?id=5123 (Issue 
raised in May 27) 

5) Cant delete tasks from Backends:

http://groups.google.com/group/google-appengine-python/browse_thread/thread/f1f674891cb5a2c1/9895847d1525d9e1#9895847d1525d9e1

Thanks,

Andrius

[Andrin von Rechenberg]

Hi Andrius

I completely disagree with your:

Lets fill in this form collectively and get things moving rather than being ignored

We have this mailing list, we have stack overflow, we have office hours

with the AppEngine team. I've never had a problem getting an answer

from the AppEngine team when I asked the question using the right

way (Mailing list / Issue tracker / Office hours).

Btw, is your thing a petition or a questionnaire? 

If it's a "petition", then it doesn't really make sense and you should just

make use of the competition (Amazon, Rackspace, whatever...).

If this is a questionnaire and you really care about helping the GAE

team to improve the service, then your approach of:

"Who is really angry with GAE team should answer these negative

questions" won't be really helpful. It wont be representative, it will

just be a list of rants and mostly unqualified because they are emotional.

I'd be more than happy and thankful to fill out an "Help improve AppEngine"

questionnaire that asks diverse questions in a neutral way and I'm sure

GAE team would be very interested in the results and will listen carefully

if you present these real results rather than a mail full of unqualified rants.

Cheers,

-Andrin, not to be confused with Andrius

PS (again): Sending lists of unanswered threads to the list doesn't work.

File bugs in the issue tracker. GAE team can't look through all threads

and see if they are resolved. That's why they offer us an issue tracker.

[Andrius A]

Hi Andrin,

you are right, maybe "unhappy" is more emotional expression, but I cant find any other word to express it in a nice way probably because my English is not so rich :)

>"Who is really angry with GAE team should answer these negative questions"

I didn't say that! I don't even talk about the team  nor blaming them in the questionnaire (you are right, not petition!), read what it says:

Purpose if this form is to found out how many of you are "unhappy" with GAE platform and why, so we could measure what is the scale and find out the main issues and when collectively report to GAE team.

I really appreciate what they did so far, their support here is great but you need to be lucky to get the answer, my point was to find people in the similar situation

I was watching this mail group and found many negative emails, and in most cases they are ignored by google, so my approach was a bit different.

So if you are happy with everything don't even bother writing here, save your time, better have a cup of tea or go and enjoy the sun!

Rgds,

Andrius

[Andrius A]

Well, it appears I am the only one here unhappy with the issues who finds time to write them here.  I cant believe GAE is out of preview and we are paying 10x higher prices and confirmed by Google that GAE admin panel is not reliable and of course they are working with that, just don't undertstand why it took so long and now they blame MS everywhere they can.

To make the things worse for you Google some of the developers here were contacted by a few major online magazines and Gartner, they all are concered with the issues we were moaning here all the time so I really hope upcomming articles will make you rethink.

[Simon Knott]

Andrius,

I'm curious as to what changes you've made to your application to fit in with the new pricing model?  

Whilst a lot of people said they were facing 10x price increase (some were 100x) when the model was first announced, a lot of those people have since come back and stated how they've managed to reduce that to 1.5-2x by re-writing their code.

Cheers,

Simon

[Andrius A]

Did you Simon reduced the costs by 1.5-2x? I dont know what are you building but if I had a blog site running in GAE I wouldn't be complaining. 

I think I am pushing GAE to the limits, my sites do a lot of ajax requests with some processing on the server (1 request per user each second) and I have thousands of users, you would say that I should use channels? Yes, I thought so, but can't since there is no way to broadcast the same message to all users and looping through each channel will delay sending message! You would then say use backends to off load sending message through backends? Ohh yes, but backends do not support channels! So backends are really good, you can have them running on background and doing time consuming work which actually saves your money! But wait, they are not reliable, they restart few times per day and this is what google says in documentation, so you need to build some mechanism on top to make sure when backends restart your app do not collapse!

So far I have moved all processing to backends and made all data to be passed to user requests via memcache and believe me this is where it gets more complicated. You later realise its nearly impossible to have backends and frontedns working in local environtment, so develpment process and debugging is really really painful. Also backends in production restart few times per day so reliability is very low, so on top you build  few master backends who look after each other and launch other backends when needed through task queue, and let this make more complicated, we know that task queue can execute in some occasions the same task few times (which is more frequent to happen within backends) and then you rely on memcache, and lets make it a bit more complicated - you know that memcache can be  flushed any time, so then you have datastore which can also time out and then to be it the most complicated to handle - request can occasionaly  fail to load. Do you see the point? You have a so many API's, we are being charged for them but they are too unreliable, thats totoally not right. I understand that nothing is  reliable, anything could happen, but at least backends should not restart few times per day, datastore shoudn't time out every hour, requests shouldn't fail every day! and finally I shouldn't be paying for something called experimental!

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/A7uwTwRi69wJ.

[Jeff Schnitzer]

It sounds like you would be pushing any architecture to the limits.  A few thousand requests per second with shared state is a nontrivial problem, especially if you have a poor read/write ratio.  If you have a long-polling scenario you may have chosen the wrong platform; this is something that node.js or twisted python or equivalent asynchronous architectures excel at.  But then expect to do a lot of work setting up and maintaining a fleet of servers; it isn't a one-man job anymore.

What percentage of your (present) cost is instance hours vs datastore operations?

> and finally I shouldn't be paying for something called experimental!

Maybe you shouldn't be *using* something called experimental.  You're utilizing GAE in a way that is not efficient.  Possibly your application is one that cannot be run efficiently on GAE; it sounds like an async appserver would be more appropriate.  But maybe it can be done.  Perviously Google hid the real cost of your app from you; now you're getting charged based on a closer approximation of the "real cost" of service.

If you want help fitting your application to GAE's architecture then you need to post a lot more details about what you are doing.  This post was the first time in your countless messages that you included enough information for any of us to even get a glimpse.  Seriously, we (the community) are willing to help, but you need to actually want help.

Jeff

--

We are the 20%

[Andrius A]

We are running auction sites in GAE, they have timer and with each user bid price increases. It deals with multiple bids by using sharded counters in MS datastore, because its much more faster and consistent. Auction data processing is done by backends and saved into memcache which is read by frontends via ajax requests.

About 60-75% is paid for instances, before new pricing this kind of site in GAE was cheap to host.

Big pain here is various issues with GAE inside, especially  requests failing to load, often datastore read deadlines, unexplained requests latency within GAE, all backends suddenly restarting in the same time..

How can you build something reliable in GAE by using these api's all together? backends + taskqueue + memcache + datastore? Try to fit in your mind, because I can't - any of them can fail any time, tasks can be executed multiple times in the queue, memcache flushes any time and datastore reads die? 

And again, errors within GAE are too frequent, services die too often, requests suddenly break or become too slow, admin console works like a snail and crashes too often.

[N. Rosencrantz]

About the cost I completely agree with Simon Knott that bad system design can cause creeping costs you eg. having a cron job every 5 seconds for what can be batched in a daily cron job would significantly increase your costs at no particular gain in efficiency so I too think that some of us who saw creeping costs and creeping CPU used the GAE environment in a wrong way and we had to unload some of our scripts such as cron jobs every 5 seconds not thinking about number of reads/writes/creations.

Which was good since the pricing force us the write program cost-efficiently - otherwise we can't afford running the apps.

So I think focus on a particular technical problem and solve it rather than bringing everything up under "unhappiness".
Work for a specific issue, debug it, make a proposed solution while you wait for the solution and then you'll have 2 solutions.

I found I could solve all problems with app engine though in a python way sometimes there seems to have been one way only to solve it well (which was webapp2 + wtforms + jinja2 for me.)

But I like that you bring this up since I too have been frustrated with lack of "real" system functions such as eg. backup where phpMyAdmin you just push the button and you get a backup in zipped SQL of your whole database and no need to worry. App engine severely lacks a good backup system, we have to write our own backup system and this is not application development, it's developer tool development which is OK and fine solving it once per environment. Solving same problem for every app (backup, file system, static files, upgrading to python 2.7 with all its migrations..) should be done once for the whole platform like making one good backup function that works for the whole platform or just an XML export of the datastore if they can't other format.

My bottom line is that I came here to do application development and found myself implementing development tools such as backup and file system.

That said, Google App Engine has enough advantages to still be the best platform and the best development environment. I just wish it was a real system that came with an export / import function and backwards compatibility and lots of ready-made builtin solutions for CMS with plug-ins etc like regular LAMP has. Regular LAMp has, to be fair, at least 5 years head start of app engine so to be fair we have to wait some time for app engine to mature and all problems will have been solved. In 5 years time, I believe app engine will still exist and that nobody more worries about their python 2.6 to 2.7 migration.

[Jeff Schnitzer]

On Tue, Dec 6, 2011 at 1:00 PM, Niklas Rosencrantz <nikl...@gmail.com> wrote:

eg. backup where phpMyAdmin you just push the button and you get a backup in zipped SQL of your whole database and no need to worry.

I found backing up a production mysql system with significant data size to be a huge pain in the ass.  Backups invariably ended up locking large sections of the database and freezing the frontends for unacceptable lengths of time.  The only solution was to set up a slave and run all backups off of the slave.

InnoDB's awful locking policy is why I will never run MySQL in production ever again.  If I need an RDBMS in production, I'll use Postgres.

This brings up an interesting question.  Without a full-database MVCC system, how do you backup the whole database?  Especially with multigroup transactions, there's no way to guarantee an isolated snapshot.  Depending on how your application works, getting a consistent backup might be impossible.

Jeff

[Barry Hunter]

On Tue, Dec 6, 2011 at 7:00 PM, Jeff Schnitzer <je...@infohazard.org> wrote:
> On Tue, Dec 6, 2011 at 1:00 PM, Niklas Rosencrantz <nikl...@gmail.com>
> wrote:
>>
>>
>> eg. backup where phpMyAdmin you just push the button and you get a backup
>> in zipped SQL of your whole database and no need to worry.

The idea of downloading a multi-gigabyte backup with phpMyAdmin, is
kinda funny.

>
>
> I found backing up a production mysql system with significant data size to
> be a huge pain in the ass.  Backups invariably ended up locking large
> sections of the database and freezing the frontends for unacceptable lengths
> of time.  The only solution was to set up a slave and run all backups off of
> the slave.
>
> InnoDB's awful locking policy is why I will never run MySQL in production
> ever again.  If I need an RDBMS in production, I'll use Postgres.
>
> This brings up an interesting question.  Without a full-database MVCC
> system, how do you backup the whole database?  Especially with multigroup
> transactions, there's no way to guarantee an isolated snapshot.  Depending
> on how your application works, getting a consistent backup might be
> impossible.

... without a readonly period - ie downtime.

Even youtube, goes readonly sometimes... (like right now)
http://www.google.com/support/youtube/bin/answer.py?answer=1751921&topic=16550

>
> Jeff

[Kaan Soral]

Can't agree with you more, you have my full support

I can't believe this last month either, I paid 1k$-2k$ extra just because 1.6.1 didn't roll out and scheduler works for Google Accounting rather than us