Temporary storage of data bigger than 1 Mega

[Shachar Grembek]

Hi,

I have a resource file that its size is 1.6 Mega (A Machine learning model). The data is required and needs to be loaded for every API call I receive, but loading the file to memory takes around 8 seconds. So I would like to find a way to have the data available in a more efficient manner. I considered to store the data in Memcache, but Memcache objects are limited to 1M. I would really appreciate, if someone can help me find a way to store it in a manner so it can be loaded it in 1 second or less.

Thanks!

[Justin Beckwith]

While I'm sure there are other options you should explore first, one idea is to look at using App Engine Flexible.  The instances in flex tend to last longer [o(days)], and the data could be cached on the instance.  

Depending on the APIs and services you're using, a change like this could be pretty extreme, so you may want to read up on the differences here:

https://cloud.google.com/appengine/docs/the-appengine-environments

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/eae5b849-f1db-43f5-af88-424a6b04890b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Justin Beckwith | Google Cloud Platform | @justinbeckwith | http://jbeckwith.com

[Shachar Grembek]

Thanks for the suggestion, but as you mentioned, it would be too extreme to redesign the architecture, just because of that

On Monday, April 24, 2017 at 6:53:24 PM UTC+3, Justin Beckwith wrote:

While I'm sure there are other options you should explore first, one idea is to look at using App Engine Flexible.  The instances in flex tend to last longer [o(days)], and the data could be cached on the instance.  

Depending on the APIs and services you're using, a change like this could be pretty extreme, so you may want to read up on the differences here:

https://cloud.google.com/appengine/docs/the-appengine-environments

On Mon, Apr 24, 2017 at 8:25 AM, Shachar Grembek <shachar...@gmail.com> wrote:

Hi,

I have a resource file that its size is 1.6 Mega (A Machine learning model). The data is required and needs to be loaded for every API call I receive, but loading the file to memory takes around 8 seconds. So I would like to find a way to have the data available in a more efficient manner. I considered to store the data in Memcache, but Memcache objects are limited to 1M. I would really appreciate, if someone can help me find a way to store it in a manner so it can be loaded it in 1 second or less.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/eae5b849-f1db-43f5-af88-424a6b04890b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick (Cloud Platform Support)]

While using the Flexible Environment or a dedicated Compute Engine instance would probably be the simplest solution, you may also consider (if committed to using the Standard Environment of App Engine) breaking up the file into several pieces are storing these in Memcache, maintaining their freshness as necessary (changes to the underlying model, a given file part being evicted from the cache).

Cheers,

Nick
Cloud Platform Community Support