Least Expensive Way to Delete Entities

[Patrick Twohig]

Hi,

I have a bit of old data that I want to clean out.  I tried to write a fanout operation to clean out some of the old data, but it's actually manged to run out my budget in a few hours.  I'm curious if there would be a way to delete huge amounts of old data that isn't needed anymore.  Is there a tool or something that i can use to efficiently purge entity groups from my application?  Any help would be greatly appreciated.

Thanks,

Patrick.

--
Patrick H. Twohig.

Namazu Studios
P.O. Box 34161
San Diego, CA 92163-4161

[Ikai Lan (Google)]

One thing you can do is making this really slow. The free quota actually gives you quite a bit of quota (if your app is relatively small) so if you can stretch out your job you can purge a relatively small datastore at low cost.

I suspect what makes this expensive is datastore ops, not instance hours. Deleting an entity with 10 properties is 21 writes! There's not a lot we can do there ...

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[supercobra]

Could you explain (again?) how a deletion of an entity w/ 10- property is 21 writes?

[Brian Quinlan]

I think that Ikai made an off-by-one-error: it requires 22 writes
assuming that every property is indexed :-)

Check out "Entity Delete (per entity)" at:
http://code.google.com/appengine/docs/billing.html#Billable_Resource_Unit_Cost

Cheers,
Brian

[voscausa]

You van use a daily cron job to delete parts of the old stuff and stay within your quota. In this way you can make use of your free new quota every day. Maybe it is not cheaper, but you stay within tour quota.

And ofcourse use batch delete.

[Ikai Lan (Google)]

Guess I forgot about the "EntitiesByKind" index:

http://code.google.com/appengine/articles/storage_breakdown.html

Yes, the correct number should be 22 datastore writes.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[SP]

Hi Ikai, 

In traditional RDBMS like Oracle, MySQL, there is truncate command to truncate a table. It is very light in term of resource usage.

Do we have similar feature to truncate a kind in datastore? Such a feature would make housekeeping old data a lot more efficient. The lack of such feature would be a major bottleneck that prevents us from moving our serious business apps to App Engine. 

Regrards, 

SP

[Jason Collins]

+1, this feature is really needed.

For us, truncation by namespace works best, though truncation of an entire Kind would also be a useful dimension. 

We're happy to "soft-truncate" the data - i.e., simply have it marked for deletion so that we can piggy-back on some other background process that is inexpensive for Google and have it finally be deleted days/weeks later.

j