Where to store thousands of Constants?

[YuRen Lin]

Hi, all

I am in the game industry and use Google App Engine as my backend server. My problem is we do the game logic calculation in the server side and use thousands of Constants which read from csv files. Our current solution is to read CSV files and store them into the datastore. Then when a request comes, the servlet checks memcache first, if miss read from datastore. This is done automatically by Objectify. I am wondering if there is other more effective way to do this.

Other solutions include:
1. store all constants in the servlet as int array. But I think the launch overhead may be a lot, since there are thousands more constants.
2. read constant from csv files each time the servlet needs it. I don't think this is a good idea.


Thanks in advance. First time to post here.

[troberti]

We also use App Engine for our game content, and in our case, all our game data is just stored in Python modules. In 'code' as you will. An advantage of this is that you can easily send game data back just from instance memory.

For us the increase in startup time is negligible, although I do not have experience with Java. My impression would be that this approach should work fine for few thousand constants.

[Jay]

One approach that can work really is to store this data as a json document in google cloud storage. This may require architectural changes however for the client to get the information.

[Nicholas (Google Cloud Support)]

Thanks for sharing your question and situation here.

From what I understand of your scenario, accessing a large array of integers from memcache and occasionally Datastore is not quite as fast as is needed for your implementation.

The fastest as you suggested would be to have all constants loaded within instance memory.  There would be a few different points to consider if using this route that regard performance:

• Compile time
• Start-up time
  
• Read time
  

Compile time

This cost is only paid once and is not incurred by your end users which I believe is your main concern.

Start-up time

This may be longer if hard-coding all your constants into your Java code.  This would still only be paid when starting instances and may not be all that significant.  I might suggest testing start-up times with some logging to really know what the time cost would be.  This time however, will not affect too many end-users barring the occasional ones to be waiting for instance to be available.

Read time

This should be very fast with an int array and still quite fast with a HashMap.  Both of these would have a negligible memory footprint if existing once per instance (as opposed to once per request).

If you noticed start-up time being a problem, you could instead store the constants in a file accessible to your App Engine service using <resource-files> or a Cloud Storage bucket and only load it upon the first request.  For example in pseudo-code:

HashMap constants = null;


private void loadConstants() {
  // retrieve values from constants.json
  // and set to constants HashMap
}


public void doGet(...) {
  if (constants == null) {
    loadConstants();
  }
  // serve response
}

The trade off here is that the time cost of getting the constants into instance memory is only paid during the first request.  You should also consider deferring it to a warm-up request as they are excellent for these sorts of tasks.

Hope this helps!

[Jeff Schnitzer]

Just a few thousand constants? Even if each was 1k (!), you’re talking about a few megabytes of RAM. Why not just load them from CSV into RAM and keep them there? 

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9046e0b0-68d3-4337-bdd6-d681ae15b028%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[YuRen Lin]

Hi, Jeff.

Do you mean load all of them into "instance memory"? I am not sure how much instance memory I have in app engine. That should work, but need to measure the startup time of instances. When a lot of requests come in, each instance has to keep these copy of constants in their instance memory.

Jeff Schnitzer <je...@infohazard.org> 於 2016年6月10日 星期五寫道：

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/Ap8z5OsgiFg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uiy3JKukPDMZLnABNdUv0ZnM2WWXRKz%3DWqb-Bi5W_tQTA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

----

YuRen Lin

Skype: yuren892

QQ: 2254642234

[YuRen Lin]

Hi, Nicholas

Thanks for your explanation. Very nice. I think the most simple and fastest way is to store all these constants in resource file and then load them to instance memory. Do you know how much instance memory a Java App Engine can hold?

'Nicholas (Google Cloud Support)' via Google App Engine <google-a...@googlegroups.com> 於 2016年6月10日 星期五寫道：

--

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/Ap8z5OsgiFg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/addc1755-0d84-4018-bec3-faae4f3f18c9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alex Martelli]

On Thu, Jun 9, 2016 at 6:50 PM, YuRen Lin <yure...@gmail.com> wrote:

Hi, Nicholas

Thanks for your explanation. Very nice. I think the most simple and fastest way is to store all these constants in resource file and then load them to instance memory. Do you know how much instance memory a Java App Engine can hold?

Depends on your instance class -- per the intro at https://cloud.google.com/appengine/docs/java/an-overview-of-app-engine , you can get up to a GB by configuring your instance class as F4_1G (if you choose the default scaling mode, automatic scaling; with manual or basic scaling, B4_1G)  though defaults are way smaller.

Alex

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADQfWyZkYJVTHVbBObeKfCFMp5K6dLZaFdt8LN%2Bs6EXM8xKCQA%40mail.gmail.com.