Worried about performance of an HTTP based game.

[Zippoxer]

I'm working on a game and that game will have an update that will run every 10 seconds.

In this update every player's data is fetched from the datastore, processed by my code (changed) and saved to the datastore.

My worry is how long this update would take if there are 2,000 players in the datastore. I'm almost sure this would take above 30 seconds, especially because I use Python. But I may be wrong in my expectation or in the method of the update.

Thanks for reading.

[Branko Vukelic]

Why not use task queue for that?

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/XuVreG-Uld8J.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

--
Branko Vukelić
bra...@herdhound.com

Lead Developer
Herd Hound (tm) - Travel that doesn't bite
www.herdhound.com

Love coffee? You might love Loveffee, too.
loveffee.appspot.com

[Zippoxer]

But how will it help me? It will still take the same time.

I asked for a way to make the update process (which I described above) quicker.

[Brian Davenport]

Why not store the user data in memcache and work with that if you can?  Followed by a periodical push into the datastore.  Have your list of users that need to be updated in a memcache key as well.

I am doing this with a forum I am kicking about, storing the thread text in a memcache key, and having a memcache key store a list of threads, when something changes it updates the keys.  I also fire off a datastore write.  When there is a miss on the key it pulls from the datastore, shoves back into memcache, and on to processing.

Brian Davenport

On Fri, Jul 15, 2011 at 09:19, Zippoxer <zipp...@gmail.com> wrote:

But how will it help me? It will still take the same time.

I asked for a way to make the update process (which I described above) quicker.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/SZ0bhx1As8MJ.

[Zippoxer]

That's a great idea, thanks :)

[Simon Knott]

The task queue option is a good one - task queue requests have 10 minutes to run, not 30 seconds.  Combined with Brian's suggestion below, you shouldn't run into any issues.

[Raymond C.]

If the "2000 players" data can fit into one entity you may use one entity to store all players' data.  Otherwise, I would say what you want to do is not suitable to be done on AppEngine, which the datastore designed to scale but not fast and write often. (in your case which is 2000 record updates every 10 seconds)

There are definitely workarounds and tricks you can employ to workaround all the AppEngine's limitation.  However, if you have not started yet, you should definitely consider to do it on other platform first because it would be hundred times easier.  Especially you are using Python which the future is not certain yet on AppEngine.

[Geoffrey Spear]

On Jul 15, 12:28 pm, "Raymond C." <windz...@gmail.com> wrote:
>  Especially you are using Python which the future
> is not certain yet on AppEngine.

Right, putting the BDFL on the App Engine team is a clear indication
they're planning to drop Python support.

[Robert Kluin]

If you've designed your data model properly writing 2000 entities
every 10 seconds probably won't matter much.

+1 to task queues and / or cron.

On Friday, July 15, 2011, Raymond C. <wind...@gmail.com> wrote:
> If the "2000 players" data can fit into one entity you may use one entity to store all players' data.  Otherwise, I would say what you want to do is not suitable to be done on AppEngine, which the datastore designed to scale but not fast and write often. (in your case which is 2000 record updates every 10 seconds)
> There are definitely workarounds and tricks you can employ to workaround all the AppEngine's limitation.  However, if you have not started yet, you should definitely consider to do it on other platform first because it would be hundred times easier.  Especially you are using Python which the future is not certain yet on AppEngine.
>
>
>
>
>
>
>
>
>

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/rMAaRbmF6g4J.

> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

--
------
Robert Kluin
Ezox Systems, LLC

[speedplane]

Another way to speed it up is to use the mapper api. It is ideal for
going through every entity one at a time. Additionally, it will have
the benefit of automatically parallel-izing the workload across
instances for you and it should scale nicely from 2000 entities to
millions (in theory).

I'm not sure whether mapper was intended to be run periodically like
you want, but I would give it a look.

On Jul 15, 4:38 pm, Robert Kluin <robert.kl...@gmail.com> wrote:
> If you've designed your data model properly writing 2000 entities
> every 10 seconds probably won't matter much.
>
> +1 to task queues and / or cron.
>
>
>
>
>
>
>
>
>
> On Friday, July 15, 2011, Raymond C. <windz...@gmail.com> wrote:
> > If the "2000 players" data can fit into one entity you may use one entity to store all players' data.  Otherwise, I would say what you want to do is not suitable to be done on AppEngine, which the datastore designed to scale but not fast and write often. (in your case which is 2000 record updates every 10 seconds)
> > There are definitely workarounds and tricks you can employ to workaround all the AppEngine's limitation.  However, if you have not started yet, you should definitely consider to do it on other platform first because it would be hundred times easier.  Especially you are using Python which the future is not certain yet on AppEngine.
>
> > --
> > You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> > To view this discussion on the web visithttps://groups.google.com/d/msg/google-appengine/-/rMAaRbmF6g4J.

> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.

[Brandon Wirtz]

Why do you say Python’s Future is unclear on GAE?  I have seen no indication of this.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/rMAaRbmF6g4J.

[Tim Hoffman]

Another alternative to consider is using a permanent backend to manage persistent game state.

That way it only needs to write out changes and not keep reading in all the records, (apart from new users joining etc).

Just a thought.

T

[Raymond C.]

In theory it would work, but in reality how much it would cost?

An typical entity write to datastore cost sth like 100ms on M/S which would be even higher for HR.  2000 entity means 200,000/ms.  Therefore if it would be done every 10 seconds, it would occupy at least 20 always active instances just for the DB write operations (in theory).  Together with game logic and communication with front end servers the number of instances that the game will need would be a lot higher.  I dont think the cost that involved is flexible on AppEngine.

[Raymond C.]

I just dont know how much my python apps will cost on AppEngine under the new pricing.  Yes, they promised new scheduler, they promised python concurrency, but we dont see anything yet except just a new pricing model that will not be good for python apps.

[Tim Hoffman]

Just because that stuff hasn't been announced yet doesn't mean it won't be.

Many things in gae land have had long gestation periods.

I have every confidence is googles ability to deliver on appengine.  Anyone who has been here from the beginning knows somethings take longer than 

we would like, but slowly but surely they do deliver.

T

[Nick Johnson (Google)]

Hi,

The easiest way to do this is to not do it, if you'll forgive the misdirection. What is the nature of the update you need to do? Is there a way you can avoid having to update every record, or to update only modified records? What is the upper-limit on players per game, and do the updates for one player depend on what the other player is doing?

It's effectively impossible to offer useful suggestions without more concrete details about what you're actually doing.

-Nick Johnson

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/XuVreG-Uld8J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Nick Johnson, Developer Programs Engineer, App Engine