HRD uses more datastore writes?

[Jason Collins]

I had a M-S app that was very regularly, daily, using around .73M
datastore writes for a daily cost of around $0.68.

I used the automatic migration tool to migrate this app to HRD.

Now, it uses around 1.44M datastore writes for a daily cost of around
$1.39 (or roughly double).

Is this correct? Does the HRD really consume double the number of
writes?

If so, given that the cost of HRD was dropped to "be the same as M-S"
to encourage people to migrate, it's rather unfortunate.

It could be that there's something else I'm not considering...

j

[Alfred Fuller]

There is no difference between M/S and HRD with regards to write ops (i.e. given the same input, M/S and HRD report the exact same amount of writes). Something else must be different (traffic, existing entity values, etc).

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jason Collins]

I'll keep looking, but it's weird because that app was consistently
within 1 penny day-to-day before, and its consistently within 1 penny
day-to-day now - it's just exactly double the overall cost...

j

On Oct 18, 2:46 pm, Alfred Fuller <arfuller+appeng...@google.com>
wrote:

> There is no difference between M/S and HRD with regards to write ops (i.e.
> given the same input, M/S and HRD report the exact same amount of writes).
> Something else must be different (traffic, existing entity values, etc).
>
> On Tue, Oct 18, 2011 at 12:46 PM, Jason Collins

> <jason.a.coll...@gmail.com>wrote:

[Will]

Jason,

Where is the automatic migration tool? I looked all places but
couldn't find it. Is it the "Copy to Another App" button on the
"Datastore Admin" page?

Thanks,

Will

On Tue, Oct 18, 2011 at 12:46 PM, Jason Collins

[Jason Collins]

I believe the tool is still in Trusted Tester phase.
j

[Johan Euphrosine]

You can sign up for the migration tool there: http://goo.gl/3jrXu.

Hope that helps.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[Will]

Thanks, Johan.

[Jason Collins]

This is making me crazy. I just cannot find the difference in the apps
(M-S vs HRD). The traffic patterns appear to be the same. The total
number of entities is very similar.

When I migrated from M-S to HRD:

- Using the current billing, the app went from $0.00 / day to $1.19 /
day.
- Using future billing, the app went from $0.67 / day to $1.40 / day.
- Datastore writes went from 0.72M / day to 1.45M / day.
- All of the cost difference is attributable to datastore writes.

There is almost no variance day-to-day with these numbers. The app has
basically zero public traffic and does the same cron-based processing
every day. The change coincided _directly_ with my migration to HRD.

_Any_ ideas?
j

[Diego Ariel Fejgelis]

Are you sure the writes are the same in both datastores?

When looking here[1], gives me the impression that HR cost is bigger than MS.

[1] http://code.google.com/appengine/docs/python/datastore/hr/

[Simon Knott]

I believe those costs are based on the old model, where datastore reads/writes were charged on a CPU basis.

It sounds like Jason's increase is in actual write counts, which is relevant for the new pricing model.

[Alfred Fuller]

Wait, are you comparing the old pricing model or the new pricing model (since you attributed the cost to writes I just assumed you were talking about the new pricing model)? In the old pricing model HRD writes cost more (~3x). 

j

[Simon Knott]

He seems to be comparing the number of writes:

 - Datastore writes went from 0.72M / day to 1.45M / day

This is the new model isn't it? 

[Renzo Nuccitelli]

I didn´t know it was just to testers. I have sent the mail when they
offer it to test and I have migrated my app to HRD too. Great tool.
Only blobs from blostore were not migrated, but i could move around
this problem.

[Jason Collins]

My datastore writes comparison (0.72M/day to 1.45M/day) is using data
all from the "new pricing" portion.

Somehow, my datastore writes doubled precisely on the day that I
migrated to HRD. My new pricing costs correspondingly doubled from
$0.67 to $1.40.

While there may be no difference in the number of writes between M-S
and HRD, is there any possibility that there is a problem in the
accounting?

j

[Stephen]

On Tue, Oct 18, 2011 at 8:46 PM, Jason Collins
<jason.a...@gmail.com> wrote:
> I had a M-S app that was very regularly, daily, using around .73M
> datastore writes for a daily cost of around $0.68.
>
> I used the automatic migration tool to migrate this app to HRD.
>
> Now, it uses around 1.44M datastore writes for a daily cost of around
> $1.39 (or roughly double).
>
> Is this correct? Does the HRD really consume double the number of
> writes?

Stray indexes?

If you haven't deleted the old app yet you could compare the old and
new using something like this: https://gist.github.com/715284

[Ikai Lan (Google)]

Long shot ... any chance your old app is still running? If so, could task queues/cron jobs on the old app might be hitting the new app and causing an increase in writes? Of course ... you would see additional load if this were the case.

This would be much easier to solve if we knew which handlers were generating additional writes, or if you were just seeing more load.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Jason Collins]

The old app is disabled.

It's tough to compare the load because the dashboard is long since
emptied on the old app. However, the number of Front-end Instance
Hours is effectively the same between old app and new app: around 24
instance-hours.

So we have:

- migration via automated migration tool
- redeploy code from old app on new app
- match Application Settings between old app and new app
- disable old app
- datastore write ops (in billing reports) are exactly double on new
app, starting exactly the day of the migration
- instance-hours are the the same between old app and new app
- (new pricing model) costs have doubled between old app and new app,
due entirely to doubled datastore write ops

j

On Oct 24, 4:39 pm, "Ikai Lan (Google)" <ika...@google.com> wrote:
> Long shot ... any chance your old app is still running? If so, could task
> queues/cron jobs on the old app might be hitting the new app and causing an
> increase in writes? Of course ... you would see additional load if this were
> the case.
>
> This would be much easier to solve if we knew which handlers were generating
> additional writes, or if you were just seeing more load.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>
> On Fri, Oct 21, 2011 at 10:30 AM, Jason Collins

> <jason.a.coll...@gmail.com>wrote:

[Jeff Schnitzer]

I really have to wonder if somehow you've added extra indexes during the migration.  That could easily cause this doubling.  In the absence of any business logic changes, this is what I would focus my investigation on first.

Jeff

[Jason Collins]

I agree Jeff, it really seems this way.

Via the console, I don't see any difference in the indexes, however,
I'm not sure what's going on under the hood. I'm hoping that someone
at Google can look for me?

I imagine I can use the cost on the RPCs (a la https://gist.github.com/715284)
to see if there is a difference between similar RPC calls. I would
almost bet that they will be coming back double.

I'll try to gather more information from my end.
j

On Oct 25, 12:34 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
> I really have to wonder if somehow you've added extra indexes during the
> migration.  That could easily cause this doubling.  In the absence of any
> business logic changes, this is what I would focus my investigation on
> first.
>
> Jeff
>

[Jeff Schnitzer]

Keep in mind that the console only shows you multi-property indexes defined in yaml/xml.  It doesn't show single-property indexes, which can be configured on an entity-by-entity basis.

Jeff

[Max Ross (Google)]

Jason,

You said that you deployed the same code to the HRD instance of your app that was running on the Master/Slave instance. Is there any part of your application that might behave differently in the face of HRD's slightly different read consistency guarantees? Are you ever writing data and then immediately querying for it with a global query under the assumption that you'll see the data you've just written?

Also, are you manually setting (lower) deadlines on any datastore calls?

Thanks,

Max

[Max Ross (Google)]

Different theory:

It looks like your cron jobs are still running in your old app. Could you please try redeploying an empty cron config for your old app and let me know when you've done that?

Thanks,

Max

[Jason Collins]

The old app is disabled. Would the crons still be running?

[Jason Collins]

Wow. So it _does_ look like the old crons were running even though the
application was DISABLED. This will likely bite other users of the
automated migration tool. You might want to address this. Note that
Ikai suggested this long ago, and I just assumed that a disabled app
would not be running cron. My fault, but a valid assumption, don't you
think?

I need to check my other disabled apps now.

I've turned off the old cron jobs and will wait for a few days to see
if there is a difference in the billing (once the billing reports
start again ;) . You'd think I'd see a difference in my instance-hours
though....

j

PS: Ironically, I had to re-enable my app to upload the empty cron
file. ;)

[Max Ross (Google)]

Cron jobs should not be running on a disabled app. We'll fix it.

Please do let us know if your datastore writes return to their original levels.

Thanks,

Max

[Ikai Lan (Google)]

Thanks for getting to the bottom of this, Jason. You're right: the cron jobs should not run on a disabled app. We'll fix this.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[Jason Collins]

Thanks all. The billing has come through now, and everything has
dropped right back to its original levels.
j

[Max Ross (Google)]

That's great to hear, thanks for being so patient with us while we sorted this out.

Max

[Jason Collins]

I see that I never responded to Max here.

Yes, once my old crons were disabled, everything returned to normal.

Thanks,
j