random result from get($key)?

[Shaojie Liu]

while we are using memcachedb as a counter backend,  sometimes we found that the get() function return a random number.

i,e, in a code segment like:
..........
$db->get('john_2009-03-4');
..........

it returns 123, in most chances.  but in 5~10% chance,  it returns 5017813, 0, or some random numbers.

what kind of problem can this be?

[Steve Chu]

Can you describe how your counter works?

--
Best Regards,

Steve Chu
http://stvchu.org

[Shaojie Liu]

///////////////
if ($db->add('key', 0))
         $db->incre('key',1);

and my problem is,  the value stored in memcachedb is correct and properly replicated to slave, but,  the return value of get() seems to be unmeaningful or random in a small chance.

[Shaojie Liu]

a new update:

when we trying loading data from memcachedb in apache,  the result tends to be random in 10+% chance.  and when we trying to reproduce with a simple script with pconnect() and get() ,  the result can be always correct.

so, could that be some issue in concurrent handling?

we use v1.0.3, db4.6 on server side, and dev-php5/pecl-memcache v3.0 on client side.

[Steve Chu]

If you are using 'add/replace/append/prepend/incr/decr', they are
atomic. But for get and set, it is not atomic, and may cause
concurrent issues.

[Shaojie Liu]

thanks for update, steve.

is there any method to get or fetch a keyvalue definitely correct from memcachedb?  supposing this is a very very basic and top important requirement i guess.

(by the way, we use intensively get/set in memcached, seems like no problem at all.)

[Steve Chu]

Whether or not it gives the correct value is up to the concurrency
model of your application. MCDB always gives the right result
according to 'set or get', but it does not guarantee the atomic
between get and set, and it can not. Seems 'cas' can simulate the
atomic 'set after get', but the cas patch has not been merged into
trunk now.

[Shaojie Liu]

our code is running in apache (200+ instances), getting and updating counter in MCDB.
we currently set two servers, one master for write, one slave for get,  dealing with 300~500 request/sec, which is not high indeed.

and the random cases happen when we merely getting counter value from slave server.

good news is that the value stored in mcdb is correct, though. i'm just wondering whether there is a flaw between php standard pecl memcache client and mcdb.

[Shaojie Liu]

btw, while viewing the release notes of mcdb, i read: 
//////////////////////////////////

2008-07-25 Steve Chu <stv...@gmail.com>
  * Bugfix: "multiple get" did not return the right value

//////////////////////////////////

so, could this be a bug in low version? (we are using v1.0.3 as mentioned above)

anyway, i'll try upgrading to v1.2.

[Shaojie Liu]

 unforunately, the upgrade doesn't work for the problem.

i'm trying to build a stress test env as:
a) write a script on production webserver,  reading data from mcdb.
     the logic is :
        new Connection();
        foreach($i=0;$i<10; $i++){
                 getValueByKey($i);
        }
b) using stress test tool on that webserver, with concurrent value varying from  5~50.

here's what we found:

•  when concurrency is relatively low,   the results are correct at most time.
•  when raising concurrency to above 20 or so,  results gone crazy. 
• etc, if a correct return is 188,171,202,144,171,124,86,22292, three kinds of wrong types:
• ,,,,2,,4,2410,188  ( some null ret. value)
• 1043,1161,1374,1055,1085,404,117698,188 (definitely wrong value)
• 202,144,171,124,86,22292,188,171 ( out of order, but with true value. weird, isn't it?)
• when testing on a devel mcdb box (storing only a few data. the production mcdb stores 1Million key-values),  it's hard to reproduce this problem, even raise concurrency to 50+.
  

i'm turning to guess whether mcdb can handle concurrency well as it is expected.

hope this makes help to debug and fix.

[Riccardo Marangone]

Shaojie Liu,

I'm interested in use of memcachedb, but the probability that in production environment I can have concurrency problem is a concern.

 

I want to replicate your problem, can you give me some more details:

- apache version

- php version

- memcache library used (version)

- kind of connection (pconnect or connect)

- timeout used in connection

- policy of memcachedb pool (if you use a pool)

 

Thanks

 

Riccardo

[Shaojie Liu]

Ric,

we'r indeed now in an intensive process of memcachedb evaluation, with steve's advice time to time.

- apache version, 2.0/2.2

- php version, 5.2.6

- memcache library used (version),  pecl-memcache-3.0.1/2 (pecl-memcached seems to be much better, but comes with new performance problem in our production env.)

- kind of connection (pconnect or connect), both, no diff.

- timeout used in connection,  default, 1 sec i guess.  tried to increase to 5 sec, problem remains.

- policy of memcachedb pool (if you use a pool), n/a.

[Riccardo Marangone]

Hi, all,

 

I can't find any conclusive post to this issue.

Have you found/fix the problem? How?

 

thanks

 

Riccardo

 

----- Original Message -----

From: Shaojie Liu

To: memca...@googlegroups.com

Sent: Monday, March 09, 2009 2:06 PM

Subject: [memcachedb] Re: random result from get($key)?

[Shaojie Liu]

by migrating php client lib from pecl/memcache to pecl/memcached.