Corrupted data in some keys, error "Memcached::get(): could not uncompress value"

[Lailo]

Hi all,

Since about 2 weeks, we have received unreproducable bug reports (i.e.
items got deleted, new items were added, game stops at initializing
etc.) from users for one of our game production which is currently in
closed beta mode. Last Thursday we realized that these are the results
from corrupted data in some membase keys. However we have no clue
what's causing this. We tried to get some more info over the weekend
and created some error logfiles, however no error was logged, but we
know that they are there :/

So while searching in the web to help our technical team, I found this
group and I hope anyone of you has a suggestion or an idea that could
help us.

This is the description from the technical team:

-------------------

When trying to get data out of membase the script throws an exception
"Memcached::get(): could not uncompress value". Following is membase
is used ($data is always a php-array)

$mb = new Memcached();
$mb->addServer('127.0.0.1', 11211);
$mb->setOption(Memcached::OPT_COMPRESSION, false);
$mb->setOption(Memcached::SERIALIZER_PHP, 1);

//LOAD a value
try
{
$data = $mb->get($key);
}
catch(Exception $e)
{
echo "Exception!!!" . $e->getMessage()." mbget $key " . $mb-
>getResultCode();
}
$result_code = $mb->getResultCode();
switch($result_code)
{
case Memcached::RES_SUCCESS:
$data = $data;
break;

case Memcached::RES_NOTFOUND:
$data = array();
break;
default:
die('mb_base load error' . $key . ' ' . $result_code);
}

//
// do something with $data
//

//SAVE a value
$mb->set($key, $data, 0);


-------------------

Thank you so much!

[Matt Ingenthron]

Hi,

Which client is in use here? It looks like perhaps pecl/memcache? The compression is usually indicated by flags. Is it somehow possible that some other part of the app has compression on?

Only once before we had a report of somehing similar and in the end it was an application level issue. Do you have any incr/decr/append/prepend operations with this data?

What does the data look like when corrupt? Do you have a hex dump or something? It may be worth comparing it to a "regular" object.

We'll try to help.

Matt

[Chad Kouse]

This can also happen if your servers are using differing versions of
the pecl extension I believe. I just saw this error last week and it's
because our servers were running version 2.0.1 and one of our devs was
running 1.0

--chad

[Lailo]

Hi Matt,

Thank you so much for your quick answer, it's much appreciated!
Memchached is in use. As far as I know there is no compression on,
also not in any other part. We have no incr/decr/append/prepend
operations with the data. I also think that we unfortunately have no
dump. I am back at the office meeting with the tech team tomorrow
morning (CEST) and will still check your answers and ideas with them
to make sure. So if any of you come up with anything, please feel free
to add your thoughts here in the meantime.

Thank you so much!

/e: @Chad, Thanks! I'll definitely check if there could be an issue
with the versions and will come back to you here with more info
tomorrow morning (CEST).

[Chad Kouse]

The pecl extension will automatically compress any value over a
certain size (unless you tell it not to). I think it's 100 bytes.

--chad

[Lailo]

Hi all,

After going through your replies with the technical team, we
unfortunately are not yet sure to know the cause.

-------------------------------------------------------------------------------------

1) Which client is in use here:
Memcached

2) Is it somehow possible that some other part of the app has
compression on?
Compression false flag was set in the end of last week. So it COULD be
that those keys that have corrupted data in them, are old keys from
the time, in which compression was still on. However we tested with
and without compression and could not reproduce an error.

3) Do you have any incr/decr/append/prepend operations with this data?
No

4) What does the data look like when corrupt?
The data is not really human readable. This is how a corrupt key looks
like:

----------------------
K^W^@^@^_a:5:{i:1803;a:31:{s:8:"horse_id"^@; :
12@^Vgender@^Z^@0@^W^@4@^W^Dbreed@4@^Y^@6`^@^Yody_colo`5^A17`^D^
\^Cmane`^A^\^@4@@8^Dsplit`^F^
\`^@ hooves`^A^]^A18@;^@3@;^Fpattern`^@'`^@7^Bleg trkings@^]`
+`^@^]^Chead@^^@^]@c^@2`^@T`,strand`^Ax@3`^A"^Ctail`
"^A11@ ^Dshiny@\^A10@^V^Dname" C!7Attila@

^@2`^@5grid_position_x@?`r`!`a^A28 Pa^AP^Forienta@A@?@\^A19@^Blif!
6rogresa^A^Y
ag oint`B 2` ;` !#` 2A ` s level@s` ` s nB3 sequenc | a:" @ a:
2B "\ typ i:B5 trea@ "-` )!@` " 43 w .amountAH!G 4";}!4 ;`\`2 2`
2 2`2 2`
2 S 5` i@` } a curr!w _ac y "3 b _ cooldowa b u tal
=!v! 6 d 20 f > V > happarbO` U` _@< 33419161A ` ^ mas#) ;` ^@\b Y
8_build#7@Lb; F } badge ! d@T!Z
show_as_foaa} 0A 24 <!@!ka * n` YdTn do ^]`^@Kpa^Ba^@
%d^Bp^@9@A`^@7dp`6`^@^]d*p` d^Vq`#b^A^]d^Vr^@1"* Mt.
MayhemB a^A{d
w^@2C`!"H !a^Ba^@Xd%wC dw^@8!Zc^AXd v@^_`^Avd^@ubSd^Qu^B0:
{#L`^AWc 0#g$

c^@Yc]/237425@xb^@hc^B/ 7 Cc"'b ^@sc'h #'dEc
'aka &h !< :"1`%/ 0h 6@#` lh 6b Qh nc Th
a @c )ac )!W Gaih 2AKb cDc" 3`{a Pc%"aca )c C{d $P a` h `Sc #
6Go'xaH D. F sub@ @Y g | 3gIfw` ?@w` ?'-(<H `?HQ`?
@Q` @ ;cK` {CY`;` ; ; 4G/` ;Aa` 5 ;hp";A!A# !i2!4'yi w#H` ' i

M#I(fe 6$ 3!g@h gB=a .e ? !6 1e; ` eB<le 1e eF? 568!Zh
=eV"+ e = + Mysti,oB a xe &C `!
eAi^Fe/ACga^A.e^A%AgB7e^SA^@8B9b^@4h^@d`ReP@(~$De1@ ?e}@e!@!e!8 ef?
^D29422bPc^@xe^B?jUw^B692bf%%c2 ^ ^_`
7`-a &ey7e a =j3t(eBlackie 5 1H e 5@!eK5B e}5e}5ei5 4187388D c we
\5 ;}}
END

----------------------

5) Do you have a hex dump?
No, unfortunately not.

6) Servers are using differing versions of the pecl extension
We have a Quality assurance server which uses the same server template
as the live environment (same versions). On the quality assurance
server we have no corrupt data, however we have it on the live
environment.

7) The pecl extension will automatically compress any value over a

certain size (unless you tell it not to).

In the end of last week, we told it not to. As written in 2) it could
be that the corrupt data is from old keys that were still compressed.


-------------------------------------------------------------------------------------

Right now we just hope that no new keys get corrupted data. We will
deploy a log to get all corrupt keys, delete those (as we don't think
that there is a way to repair the onces that have garbage in them) and
cross fingers that no new keys get created.

If it is not only for old keys (so that it was not the compression) we
will need to investigate further (i.e. in the deployment process
etc.).

I will keep you posted.

Thanks so much for your help!

[Lailo]

Hi again,

Just to let you know: We were lucky! No new errors of corrupted keys
occured in the log since Monday. We are still not 100% sure, what had
caused the issue, one of the actions we and our hosting partner took,
however seemed to be successful! :)

So in short, what they and us did was:

- Friday: Surpressing the automatic compression
- Monday: Killing two server instances, that apparently were in the
wrong availability zone (and that might have had different
configurations than the instances that are running now..)

After these actions the corrupted keys, that we knew of, were also
fine again!

This topic is therefore "solved" for us ;)

Thanks again for all your help!