Memcache doesn't work

[Jamil Rzayev]

Hi.

I have to use memcache to store some values frequently accessible by
application I am working on.
The Google App Engine Memcache doesn't work as it apparently removes
values after a couple of minutes I store it at memcache.

I really need any idea and would highly appreciate for any comments.
I have already tried to ask here, but got response that I need to
store values as soon as I read them and didn't work.

I am still losing my values...

[Brandon Wirtz]

Memcache is just a cache. Your values have no guarantee of being there even
1micro second after you did the write.

If you post your code we might be able to help, but I assure you Memcache
does work.

My app does reads from the datastore or memcache for every request,
sometimes to for initialization of variables... like Who am I, Why am I
here, what configuration am I in. And then to serve the request...

We use a combination of instance memory and memcache to not have to talk to
the data store. Datastore reads are "slow" compared to Instance Memory and
Memcache.

Datastore Write Operations 0.24 Million Ops 0.19
$1.00/ Million Ops $0.19
Datastore Read Operations 0.02 Million Ops 0.00
$0.70/ Million Ops $0.00

I didn't count the "reads in my app" to the "writes" but I know I do reads
for initialization every time, and reads for data every time. I only do
writes on some requests... So I am going to be conservative and say that I
do 3x as many reads as writes in my code, but so far today I have hit the
Datastore with a write 10x as many times as I have a read. That works out
to my local memory and memcache saving me 97% of the time.

I don't have an easy way to tell you how much of the time that is instance
memory and how many times it is memcache, but I assure you it works.

Hi.

--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengi...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

[Deepak Singh]

My application while on loading makes a read operation around 6500 rows which is fixed for app. I have set memcache for this feature with cache expiration to 8 days. Bu i observe that cache usually fails 5 to 6 times in day, so in that case my app has datastore read of around 6500 rows. Thus my app heavily uses datastore read quota whic is very costly to me.

I have been unable to make cache available for 8 days or even working properly.

It would be great if you can let me know how to optmize this cache properly.

Thanks

Deepak

--
Deepak Singh

[Jeff Schnitzer]

Memcache doesn't work that way.  You don't get any guaranteed quantity of cache and it's not guaranteed to last for any specific length of time - memcache is a shared resource and google allocates it according to their own magic algorithms.

Data falling out of the cache every 4-5 hours?  That doesn't sound like a problem to me.  If you have cache data that needs predictable durability, consider a backend.

Jeff

I am the 20%

[Deepak Singh]

ok. So if i fetch these 6500 entities through a backend and set them to cache for 8 days, will there be durability for 8 days?

My app is low trafffic app and there might be time period when there is no traffic to site setting instances to 0 for that time period.

Thanks

Deepak

[Timofey Koolin]

Backend can work with big uptime, but without gurantee stable durability - it can be restart from time to time + you must pay for full-day backend usage - from $1.2 per day for smallest backend class.

2011/11/21 Deepak Singh <deepaks...@gmail.com>

--
Blog: www.rekby.ru

[Brandon Wirtz]

If you are reading 6500 rows every request you are doing something wrong.  Why would you possibly need that much data?

[Deepak Singh]

The reason for reading is, they are the city names in INDIA and i need to display all the city names for each client. 

each entity has only two properties -cityname, cityid - but there are 6500 entities.

i am not able to find any solution for this situation.

What i am planning to do is,

Read them once and put in an Arraylist object and share this object everywhere. But how can i create one such arraylist object which lives forever and can be shared across all instances.

Pls suggest.

Thanks

Deepak

[Simon Knott]

Why don't you store this data in a single entity, as serialised data? If the data expires from the cache, you then only have a single entity read...

[Brandon Wirtz]

They never change then.

 

Put the array hard coded in to your code.

 

$MyCities = array(‘Dehli’,’New York’,’Austing’);

 

Fine that is php code and only one of those is in India… But you get the idea.  No  reason to use data store for that.

[Simon Knott]

Ha. Yes, Brandon's idea is better. Need caffeine...

[Jeff Schnitzer]

On Mon, Nov 21, 2011 at 4:39 PM, Deepak Singh <deepaks...@gmail.com> wrote:

Yes. Putting them hard coded values is the last solution but it is tough to make array of 6500 hard coded values.

Come now, that's silly.  Can you print it to XML?  JSON?  CSV?  You could even use a templating system to output pure python.  This is like a job interview question.

Jeff

[Brandon Wirtz]

Huh?

 

Way 1:

Put the list in a CSV

String = “Text of CSV”

Explode(String)

 

Way 2:

Read Data from data Store

Serialize Data

Echo/Print Serialized Data

 

Copy/Paste Serialized data in to code

 

Way 3:

Copy data from data viewer in to excel

Export to CSV

Use find and replace to replace New line with New line “   and , with “,”

Put a curly at the beginning and end, and paste in to your code.

 

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Deepak Singh
Sent: Monday, November 21, 2011 12:40 PM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Memcache doesn't work

 

Yes. Putting them hard coded values is the last solution but it is tough to make array of 6500 hard coded values.

 

@Simon,

 

How do i store them as serialised data in datastore ?

 

Regards

Deepak

On Tue, Nov 22, 2011 at 1:38 AM, Simon Knott <knott...@gmail.com> wrote:

Ha. Yes, Brandon's idea is better.  Need caffeine...

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/6rqyUnPbxIAJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

 

--
Deepak Singh

[Deepak Singh]

I would prefer way 2. Using serialised data.

I am a java developer, can you pls let me know how to achieve way 2 ?

Thanks

Deepak

--
Deepak Singh

[Deepak Singh]

Yes. Putting them hard coded values is the last solution but it is tough to make array of 6500 hard coded values.

@Simon,

How do i store them as serialised data in datastore ?

Regards

Deepak

On Tue, Nov 22, 2011 at 1:38 AM, Simon Knott <knott...@gmail.com> wrote:

Ha. Yes, Brandon's idea is better.  Need caffeine...

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/6rqyUnPbxIAJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Deepak Singh

[Andrei Volgin]

1. If you absolutely have to have all 6,500 city names for each user
session, I would put them in a separate JavaScript file, stick a
reference to it at the bottom of your app page, and host this file
somewhere outside of GAE to save a bit of money. This file will be
cached by browsers, so that returning visitors most likely won't have
to load it again.

2. Now, I very much doubt that your users click/browse/view more than
a dozen cities during their visit. I cannot come up with any use case
why you need to load all 6,500 names for each visitor. Users will
never look at more than 50 records. If you need to load/display more
than 50 city names at a time, you have the wrong data model or bad app
design.

[Deepak Singh]

In anycase i have to have these 6500 entities for each user session.

Now i want to store it as serialised data in datastore.

public class City implements Serialisable{

     private string cityname;

      private string cityid;

      // getter setter

}

now i have these 6500 entities added to a list.

List<City> list = somemethod() ;

Now this list contains 6500 object of City.

Could someone pls tell me how can i convert this list object to byte[] and store them as Blob in datastore and retrieve them back as byte[] and covert  them to list.

Help is greatly appreciated.

Thanks

Deepak

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Deepak Singh

[Timofey Koolin]

If it more than 1MB you can zip it or split to 2-3 entities.

2011/11/22 Deepak Singh <deepaks...@gmail.com>

--
Blog: www.rekby.ru

[Deepak Singh]

it is less than 1 MB. it is around 120kb.

but how to store it in datastore .

[Simon Knott]

What persistence framework are you using?

[Deepak Singh]

I am using Objectify 3.0.*

On Wed, Nov 23, 2011 at 1:32 AM, Simon Knott <knott...@gmail.com> wrote:

What persistence framework are you using?

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Dcm2VWTxzfEJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Deepak Singh

[Simon Knott]

Just annotate it as @Serialized

[Simon Knott]

Sorry, I've actually just gone back and followed your last few messages.

If you are really going to go down the route of persisting it to the datastore (and I still think that's wrong), then create a completely new Entity type.

e.g.

public class CityData {

public static final String ALL_CITIES = "ALL_CITIES";

@Id

private String id;

@Serialized

private List<City> cities;

}

Then persist that with a key shown in the static variable - then whenever you really need the data, you can just retrieve it by the same string key.

I'm still confused as to why you couldn't throw all of this data into a static file containing JSON / XML, as Jeff and Brandon suggested?

[Deepak Singh]

you mean to say

@Serialised

public class City implements Serialisable {

      private string name;

            private string id;

// getter setter

}

List<City> list = somemethod();

Objectify obj = ObjectifyService.begin();

obj.put(list);

So the above code will be able to put list object as serialised object in datastore and will be able to get back as

List<City> list = obj.get(City.class);

Am i right ?

On Wed, Nov 23, 2011 at 1:46 AM, Simon Knott <knott...@gmail.com> wrote:

Just annotate it as @Serialized

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/X7FiDN1gWMkJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Deepak Singh

[Jeff Schnitzer]

No, that won't work.

Seriously, this is all spelled out in the manual.  It took a lot of time and effort to write - please read it:

http://code.google.com/p/objectify-appengine/wiki/IntroductionToObjectify

Jeff

I am the 20%

[Deepak Singh]

Here the doc says that the object graph can be serialised with some limitations.

However i think i would be able to store the List object and get it back.

I would give it a try and see what happens.

Deepak Singh