Purging Datastore Content

[Stefan Podkowinski]

Last Saturday I decided to cleanup data in my GAE app to avoid getting billed for it in the future, as those $0.02 bills on my cc are really annoying. The app is basically dormant so what could go wrong.

So I selected a collection in the datastore admin with about 1.5m records and 500mb and confirmed "Delete Entities".

Shortly afterwards the app started spinning up instances. Also after a minute or so I reallized that the app was eating up cpu quota like crazy and deleted the tasks from task queue as soon I started to get into the billable zone.

Today, two days after, I got back to the dashboard to see the result: nothing changed at all.

And those are the numbers from the "Datastore Statistics".

From the "Quota Details" page:

Total Stored Data

15%

1.64 of 11.00 GBytes

Billing history yesterday:

Stored Data: $0.005/GByte-day

1.65

1.00

0.65

$0.01

This is exactly as has been before the datastore admin operation.

However, when I get to the datastore statistics or datastore admin the numbers did change:

Breakdown by Property Type

Property Type	Size
Integer	151 MBytes
Date/Time	62 MBytes
Blob	37 MBytes
String	12 MBytes
NULL	353 KBytes
Key	111 KBytes
Boolean	92 KBytes
Text	23 KBytes
Metadata	413 MBytes

About 675 MB (also matches with the values shown in the datastore admin).

So my first question would be is why do I get billed for more data than I store?


Next I checked how I would be doing with the new pricing model. I'd expect that this kind of task could propable be done with a single backend instance so I would not have the cpu quota problem. Again I got disappointed.

Old Model:

CPU Time: $0.10/CPU hour

8.77

6.50

2.27

$0.23

Thats $0.23 on cpu

New Model:

Datastore Writes:   $1.00/Million Ops

2.80

0.05

2.75

$2.75

Small Datastore Operations:     $0.10/Million Ops

1.46

0.05

1.41

$0.15

Thats $2.90 total billed on datastore ops. No additional instance costs though.
There have been 2.8m "writes" which makes it expensive. I deleted about 1.4m records so I'm not sure why that number is doubled, probably because of index updates?
After doing some math I figure out that it would take 725 days until I'd start saving on stored data costs for compensating the write costs involved deleting my data.
Looks like you better purge your data stores before the new pricing model goes live if you have some cpu quota left.