Sharded counters, what's the right number of shards to use?
116 views
Skip to first unread message
nischalshetty
Jul 24, 2010, 11:45:33 PM7/24/10
to Google App Engine
Incrementing a sharded counter in my app seems to fail a lot. The
particular sharded counter gets updated at the rate of 5-8 times per
second. Currently the number of shards here is 25. I'm planning to
increase it to 100 and see what happens.
I'm not sure if this is the right thing to do, if the number of shards
I'm using are low, just right or too much. If any of you have any idea
please let me know about it.
I've read that more the shards the better but what's the limit, what's
the right number where do we draw the line?
particular sharded counter gets updated at the rate of 5-8 times per
second. Currently the number of shards here is 25. I'm planning to
increase it to 100 and see what happens.
I'm not sure if this is the right thing to do, if the number of shards
I'm using are low, just right or too much. If any of you have any idea
please let me know about it.
I've read that more the shards the better but what's the limit, what's
the right number where do we draw the line?
djidjadji
Jul 26, 2010, 8:00:04 AM7/26/10
to google-a...@googlegroups.com
Are you sure the shards are in DIFFERENT entity groups. (No parent objects)
Have you looked at which shards get the most counts. Maybe your method
of choosing a shard can be better.
With transactions you can update an entity group 1/sec sustained.
With 8 updates/sec for the counter you should be fine with 10 shards
in theory, so using 20-25 shards should be more then enough.
Have you looked at which shards get the most counts. Maybe your method
of choosing a shard can be better.
With transactions you can update an entity group 1/sec sustained.
With 8 updates/sec for the counter you should be fine with 10 shards
in theory, so using 20-25 shards should be more then enough.
2010/7/25 nischalshetty <nischal...@gmail.com>:
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>
nischalshetty
Jul 26, 2010, 2:59:25 PM7/26/10
to Google App Engine
@djidjadji
Thanks a lot for the info. I had goofed up. The entity in question
that was causing updates to stack up just had 1 shard. Instead of
creating shards I ended up creating 25 entities of the same name :
( Big mistake on my part, rectified it, it's working smooth now!
-Nischal
On Jul 26, 5:00 pm, djidjadji <djidja...@gmail.com> wrote:
> Are you sure the shards are in DIFFERENT entity groups. (No parent objects)
> Have you looked at which shards get the most counts. Maybe your method
> of choosing a shard can be better.
> With transactions you can update an entity group 1/sec sustained.
> With 8 updates/sec for the counter you should be fine with 10 shards
> in theory, so using 20-25 shards should be more then enough.
>
> 2010/7/25 nischalshetty <nischalshett...@gmail.com>:
Thanks a lot for the info. I had goofed up. The entity in question
that was causing updates to stack up just had 1 shard. Instead of
creating shards I ended up creating 25 entities of the same name :
( Big mistake on my part, rectified it, it's working smooth now!
-Nischal
On Jul 26, 5:00 pm, djidjadji <djidja...@gmail.com> wrote:
> Are you sure the shards are in DIFFERENT entity groups. (No parent objects)
> Have you looked at which shards get the most counts. Maybe your method
> of choosing a shard can be better.
> With transactions you can update an entity group 1/sec sustained.
> With 8 updates/sec for the counter you should be fine with 10 shards
> in theory, so using 20-25 shards should be more then enough.
>
Reply all
Reply to author
Forward
0 new messages