Entity size and latency

[Matija]

Does anybody have some experience data (or maybe even graph) with correlation between entity size and average datastore get latency?

P.S. Also I wanted to test this new group interface :D

[Daniel Florey]

I do not have a graph but you should keep in mind that entities with large list properties may cause increased serialization time.

See:

http://youtu.be/AgaL6NGpkB8?t=14m31s

[Robert Kluin]

Last time I benchmarked there was very little, if any difference,
based on the size of an entity (ie 10kb vs 900kb).

What makes a very noticeable difference is serialization. If you've
got a 2kb entity with a 500 item list it will likely perform worse
than a 900kb entity with a single blob property -- especially if the
list is indexed. This is why I often serialize data and stick it in a
blob/text property myself when 1) I don't need it indexed and 2) I
don't need to access it every time I access the entity.

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/zNpScQ7BHaEJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Matija]

Hi Robert,

I am actually doing some analysis in task requests and before I can start I collect data in query batches with cursor. My entities are not in 900kb size, most are under 1kb, some can go up to 10kb, and some rare a little bit higher. My task latency is not so important (for completion  time) than its impact on my overall gae app and its instances. So I created 100 entities batches and questioned what is optimal number for my projected task latency in connection with entity size. Datastore infrastructure, network between datastore and app service, etc. has some constraints. Question is now is there some difference in get latency between 500 bytes or 10kb entites.

Matija 

[Brandon Wirtz]

>Question is now is there some difference in get latency between 500 bytes or 10kb entites.

 

Yes. And it seems to be about 72Megabits per second the last time we ran a bench mark.  We haven’t run it in a while. But for 500bytes and 10kb, I don’t think you’d see much difference. 900k you do.

 

We didn’t do a packet/fetch size analysis to see what unit size it is… (for example 500 bytes and 2kb could be the same speed, but 3998k and 4001k might be as much as twice the difference for the extra 1k if that means two packets, or 10% if the packets were 400 bytes

[Jeff Schnitzer]

The difference should be negligible.

While you may have to worry about serialization issues in Python, the Java GC is quite efficient.  I'll bet Go is pretty fast at that kind of thing too :-)

Jeff

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vGErIdjEKxgJ.

[Brandon Wirtz]

> I'll bet Go is pretty fast at that kind of thing too :-)

 

Did Go Support get moved to SO too?

 

[Johan Euphrosine]

On Wed, Feb 15, 2012 at 9:54 AM, Brandon Wirtz <dra...@digerat.com> wrote:
>> I'll bet Go is pretty fast at that kind of thing too :-)
>
> Did Go Support get moved to SO too?

Nop, it is still experimental so a specific group is being used:
http://groups.google.com/group/google-appengine-go
--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations