Abysmal Query Performance in Backend

[Volker Schönefeld]

Hey,

Yesterday, I've had a service running to delete a huge (1 TB) backlog of data from our appengine instance. It was working well at around 100 entities/sec (has about a million entities). Today, however, it slowed down to around 0-1 entities/sec, using the same code.

I tracked it down to queries taking very long and then failing at a rate of around 90% with a timeout exception. I've configured the deadline to None, but that didn't help. Plus, yesterday everything worked fine, I've deleted around a million entities before it started to choke.

The application ID is 'towermadness', the backend ID is 'deleter'.

Thanks!

Volker

[Volker Schönefeld]

Oh, I ment the dataset has 100 million entities, so it's far from being done. There is still around 850 GiB worth of data that wants to be deleted.

[Alfred Fuller]

The datastore does not delete things right away. It marks them as delete and waits for a "compaction" to actually remove the data. If a lot of data is deleted at the start of a query, the datastore will have to skip all the deleted rows until it finds the first real entity. This is what is causing your timeouts. 

You can avoid this problem by using cursors to continue work in subsequent requests. Cursors will cause your query to start at the last deleted row instead of the start of the query. Another good alternative is to use the Map framework (which also bookmarks it's progress for continuation between requests).

On Sun, Sep 4, 2011 at 1:52 AM, Volker Schönefeld <volker.sc...@gmail.com> wrote:

Oh, I ment the dataset has 100 million entities, so it's far from being done. There is still around 850 GiB worth of data that wants to be deleted.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/WPeRpAqa5j8J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Volker Schönefeld]

Hey Alfred,

thanks for the reply. I've got a couple more questions regarding the compaction, I hope you can answer a few:

- When do those compaction events happen? 

- What is their frequency? 

- How can I tell if one happened?

Thanks!

Volker

[Alfred Fuller]

On Sun, Sep 4, 2011 at 1:31 PM, Volker Schönefeld <volker.sc...@gmail.com> wrote:

Hey Alfred,

thanks for the reply. I've got a couple more questions regarding the compaction, I hope you can answer a few:

- When do those compaction events happen? 

The process that controls the compaction balances need with many other considerations. The fact that you have done a lot of deletes should cause it to compact sooner than it otherwise would have.

- What is their frequency? 

 

On the order of days 

- How can I tell if one happened?

Your query will suddenly start working again. 

Thanks!

Volker

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/gOrBlHV1s2EJ.