Insertion Performance

[Martyn]

I am using the low-level Java Datastore interface and writing a
"transactional" store - where each entity stores a number of data
updates in a single Blob.
Each transaction requires two updates, one to a Header entity, and one
to the new transaction data entity. Keys for each are created
explicitly. This has a number of advantages, specifically with regard
to guaranteed transaction isolation and caching potential.
However, I am disappointed with the insertion performance. I use the
DatastoreService put(Iterable) to invoke the two updates witha single
API call (or rather one update and one insert), but I am seeing >120ms
request times in the logs. The log trace confirms the delay is largely
down to the datastore put.

Am I missing something, or is this really the best I can hope for?
Apart from the blob data, which is normally ~1k, a couple of debug
properties (Date and VM hash) are included but set using
setUnindexedProperty.

Any help/advice appreciated.

[Martyn]

After some more detailed logging I have isolated the problem and it is
worrying.

It so happens that I make several memcache requests, I had assumed
that since I was making
calls to the Datastore API, any delay would be due to that, but it
turns out that around
half the delay is due to the memcache calls.

Specifically, I am seeing ~13ms latency for each memcache (single get/
put/increment), and since I am
making typically 5 calls, I am seeing ~65ms delay due to this.

This is in contradiction with AppEngine System status that indicates
memcache latency ~4-5ms.

Of course, what is meant by latency vs call elapsed is a little
ambiguous, but the figures for Datastore latency
tie up with what I see in my request logs.

In other words I time the api call/return and I see around the same
figures for Datastore latency but almost three times the memcache
latency.

Is this a problem specific to the Java API perhaps?

If I batch three memcache puts using putAll, I see 13ms for the three
rather than 39ms.

[Nick Johnson (Google)]

Hi Martyn,

The Java interface to the Memcache API, unfortunately, is noticeably
slower than the Python version. This is the cause of the discrepancies
you're seeing here. As always, using the batch interface to minimize
round trips is a good idea.

Please do file a bug, though - the discrepancy between runtimes is
something we'd like to address.

-Nick Johnson

--
Nick Johnson, Developer Programs Engineer, App Engine

[Martyn]

Thanks Nick, glad I wasn't going mad.

I have reported as issue 1962

- Martyn

On Aug 7, 4:28 pm, "Nick Johnson (Google)" <nick.john...@google.com>
wrote: