Re: [google-appengine] Help! Billing Out of Control!

[Takashi Matsuo]

Hi Stephen,

What is your app-id? Can you share the error message? I don't think anyone can help you without knowing more.

Few things came to mind though:

How big is the traffic to your app now? Doesn't your app just have a big spike?

If you're running erroneous tasks on the task queue, you can pause the queue from the admin console for a band-aid purpose.

If you have some error-handling implementations which can lead to an infinite retry loop(with or without task queue), you need to fix the implementation.

-- Takashi

On Wed, Jan 2, 2013 at 8:03 PM, Stephen Blum <blum.s...@gmail.com> wrote:

Help! We are bing charged $30/per minute!  A major datastore error.  Help!  Keeps increasing!  Help!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mCpqsBgvSkcJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Takashi Matsuo | Developers Advocate | tma...@google.com

[Stephen Blum]

Hi Takashi!

App ID: See Screenshot Below

Full Usage Stats: See Screenshot Below

[Takashi Matsuo]

Hi Stephen,

All of your tasks in the 'default' task queue are retrying more than 17 times. I suspect these tasks contribute to the high usage of the datastore.

If that's the case, maybe you can stop your 'default' queue first, and fix the settings for your 'default' queue to prevent the infinite loop(see the taskqueue settings doc[Python|Java]), and then fix the root cause, and then finally you can restart the default queue.

I also noticed that you're using deprecated M/S datastore, which is very susceptible to just a small failure in a single datacenter. I strongly recommend migrating your app to HRD.

-- Takashi

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/HO4kpXH_AAQJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Stephen Blum]

Hi Takashi,

Thank you for the details!  We are in the process of migrating now, and I expect that is cause.  However it was my understanding that the migration was a free process as per the docs.  Why is it incurring such a high charge?  Those tasks queues are old and from long ago that has not changed for about a year.  However now that we are starting the migration the billing has overblown!  It's almost at $2,000.00 for just a few hours.

[Takashi Matsuo]

On Wed, Jan 2, 2013 at 9:48 PM, Stephen Blum <blum.s...@gmail.com> wrote:

Hi Takashi,

Thank you for the details!  We are in the process of migrating now, and I expect that is cause.  However it was my understanding that the migration was a free process as per the docs.  Why is it incurring such a high charge?  Those tasks queues are old and from long ago that has not changed for about a year.  However now that we are starting the migration the billing has overblown!  It's almost at $2,000.00 for just a few hours.

Yes, our migration tool doesn't cost you for the migration process itself. If you're using our migration tool, I believe that the high usage of the datastore is caused by other reasons(it is very likely the infinite retry loop I mentioned before).

Can you tell me how you're migrating your app? If you're using any other tools for the migration, it will very likely cost you for the migration process itself.

-- Takashi

 

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/nIwnzB7DlQIJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Stephen Blum]

Hi Takashi,

Thank you again for the fast response and direct details.  You are being very helpful.  There are Datastore buttons and controls which we are using in the GAE Dashboard under Datastore Admin section.  Are we to understand that those are not honored with the Migration Path?  If this is the case it was a massive mistake that we made: May ask for a refund?  We are using these buttons (as shown in the screenshot).

[Takashi Matsuo]

Hi Stephen,

Unfortunately, you did it wrong. Please refer to the doc and migrate with our migration tool.

Re: refund

I don't think we can refund for it, sorry.

-- Takashi

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/AVBE4004J_YJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Stephen Blum]

Hi Takashi,

Thank you for the update.  How do we properly apply for errors in billing?  Not allowing us Billing Credit for a Migration Path that is Required does not makes sense!  :-)  There must be a way to correct this error.  Basically GAE Dashboard is currently:

1. Offering a Migration Button that will instantly charge $2,000.00 (why why why?) and
2. Warning loudly in a big yellow warning banner that told us to Migrate Now! and it would actually be free.

It is my understanding that we are suposed to migrate.  Yet it isn't free...?  No worries!  We won't migrate then.   :-)  That solves that part... now How may we make it so we are not charged for this FAKE OUT migration process?  Please Help!

[Takashi Matsuo]

Hi Stephen,

On Wed, Jan 2, 2013 at 10:49 PM, Stephen Blum <blum.s...@gmail.com> wrote:

Hi Takashi,

Thank you for the update.  How do we properly apply for errors in billing?  Not allowing us Billing Credit for a Migration Path that is Required does not makes sense!  :-)  There must be a way to correct this error.  Basically GAE Dashboard is currently:

1. Offering a Migration Button that will instantly charge $2,000.00 (why why why?) and

Can you tell me which button you're talking about?

1. Warning loudly in a big yellow warning banner that told us to Migrate Now! and it would actually be free.

As long as you use our migration tool, the migration process itself should be free.

 

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uHtKkpji1RYJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Stephen Blum]

Hi Takashi,

We tried using the Migration Buttons in the Datastore Admin section labeled "Copy to Another App" and "Delete Entities".  The button labeled "Copy to Another App" is totally related and relevant to migration and the way it is described in the Migration Document.  The two important notes I took from the Migration Doc were:

1. You do not incur any billing costs when you do this migration.
   
2.  you will make your old application ID an "alias" for the new one; thus, if users browse your application at old-name.appspot.com, they will instead see your new application (which you might think of as new-name.appspot.com).
   

That made it totally obvious what we needed to do.  Basically create a new App Name.  Then Copy the Data over to the new App.  This is the direction we followed and yet apparently we were surprised by a $2,000.000 bill in only a few moments. Please help us fix this billing error!  :-)

[Stephen Blum]

Hi Takashi and Google Team,

We made the transition successfully from Master/Slave to the new High Replication datastore. However it cost us just under $2,000.00 and it was supposed to be free.  Please let us know how we fix this billing error.

[Chad Vincent]

I don't mean to butt in, but it appears you didn't read the migration docs properly.  The migration buttons are under Application Settings > View Migration Tool, not under Datastore Admin.

https://developers.google.com/appengine/docs/adminconsole/migration#Using_the_Migration_Tool

[Christina Ilvento]

Hi Stephen,

I'll start a separate thread with you so we can discuss the specifics.

Thanks,

Christina, Google App Engine Product Manager

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jp1qYT_WrokJ.

[Stephen Blum]

Hi Christina,

Thank you for the follow up!  This is definitely a good start and would absolutely be helpful to get us a $1000 discount.  Please let me know if we can't approach the ~$1,900.00 which as you can tell from our daily average billing rate is much much higher and is directly related to the Master/Slave Replication transition to High Replication.  I've attached the screenshot.  I look forward to hearing back!  :-)