I can make memcached work in multi programming platforms environment
148 views
Skip to first unread message
LockeVN
Oct 17, 2008, 12:05:06 AM10/17/08
to memcached
Hi,
This is my case.
My (our) solution has several programs.
- Some websites are PHP on LAMP
- some bots are Java and run on both Windows (server 2003), Linux
(Ubuntu 8.04) box
- Some C# (.NET 2.0) tools (for content management users), run on
Windows (XP)
- Memcached server is run on Ubuntu8.04 box.
PHPwebsites put data to memcachedServer, other php websites can read
from memcached, of course. We use default PHP memcache extension.
But
- java bot can not read from memcached if value is set by PHP. But
Javabot can read If value is set by Java, on the same memcached
server. We use http://www.whalin.com/memcached/ clientAPI.
- C# tool also can not read if value is set by PHP. We use
http://www.codeplex.com/EnyimMemcached/ and http://code.google.com/p/beitmemcached/
client API . But C# tool can read, if value is set by C#, on the same
memcached server too.
The test case is very simple, no hash key, no flag, no compress, ...
Server running in the console by: memcached -p 13579 -d
PHP do: $mc->set("test","sayFromPHP");
all other do: memcacheClient.Get("test");
At the end, we found that http://bleu.west.spy.net/~dustin/projects/memcached/
Java ClientAPI can work, it can read the value set by PHP. I want to
make all programs work all together, talk memcached as common
language. But what is the problem with all above clientAPI? Are there
any trick?
Sorry for my bad English.
Thanks all.
This is my case.
My (our) solution has several programs.
- Some websites are PHP on LAMP
- some bots are Java and run on both Windows (server 2003), Linux
(Ubuntu 8.04) box
- Some C# (.NET 2.0) tools (for content management users), run on
Windows (XP)
- Memcached server is run on Ubuntu8.04 box.
PHPwebsites put data to memcachedServer, other php websites can read
from memcached, of course. We use default PHP memcache extension.
But
- java bot can not read from memcached if value is set by PHP. But
Javabot can read If value is set by Java, on the same memcached
server. We use http://www.whalin.com/memcached/ clientAPI.
- C# tool also can not read if value is set by PHP. We use
http://www.codeplex.com/EnyimMemcached/ and http://code.google.com/p/beitmemcached/
client API . But C# tool can read, if value is set by C#, on the same
memcached server too.
The test case is very simple, no hash key, no flag, no compress, ...
Server running in the console by: memcached -p 13579 -d
PHP do: $mc->set("test","sayFromPHP");
all other do: memcacheClient.Get("test");
At the end, we found that http://bleu.west.spy.net/~dustin/projects/memcached/
Java ClientAPI can work, it can read the value set by PHP. I want to
make all programs work all together, talk memcached as common
language. But what is the problem with all above clientAPI? Are there
any trick?
Sorry for my bad English.
Thanks all.
Dustin
Oct 17, 2008, 1:06:36 AM10/17/08
to memcached
On Oct 16, 9:05 pm, LockeVN <lock...@gmail.com> wrote:
> At the end, we found thathttp://bleu.west.spy.net/~dustin/projects/memcached/
> Java ClientAPI can work, it can read the value set by PHP. I want to
> make all programs work all together, talk memcached as common
> language. But what is the problem with all above clientAPI? Are there
> any trick?
Well, we don't have common object serialization for all clients.
> make all programs work all together, talk memcached as common
> language. But what is the problem with all above clientAPI? Are there
> any trick?
You pretty much have to standardize on what your objects look like in
the cache before you can expect this to work.
There's been talk of using JSON for this, but nobody's actually put
effort into doing it, AFAIK.
It'd be an easy enough plugin in my (spy) client, but would be a bit
limited in use. Most thing I end up caching can't be easily
represented in JSON (unless there's some kind of magical introspective
ORM out there).
YAML is another possibility that might be a bit easier.
Either way, it's a bit of a process. First we have to herd some
cats to converge towards a common format. Then we have to try to get
everyone else to play along. Then we have to make sure it's optional
so the native formats still work fine as they're probably more
generally useful.
LockeVN
Oct 17, 2008, 4:17:29 AM10/17/08
to memcached
Thanks Dustin again, for your working Java ClientAPI ;) .
Because of our solution has complex parts, several programs in several
programming languages, SO I want they (programs, websites, tools,
bots) talk the same language on the same channel. The common channel
is memcache, and we use native format for values, the common language
is STRING format, only string. All values we use to test are string.
We do the seriallization (simple ini-like, XML or JSON, all encode in
UTF-8) before put it to our MemcacheHelper to write to memcacheD.
@Dustin: I think you should left the seriallization jobs to whom want
to use memcache. They have several systems, they need 'em talk
together, so they know what the most suitable seriallization way to
implement, NOT YOU. They (the wild wide world who use memcache)
develop program (maybe big program) know how to serialize :D , don't
worry. Thanks again.
Memcache must operate well, fast, reliable and simple. Memcache is
useful and famous because of that feature. Add more complex
transparent general enterprise blah blah features will make memcache
become something like "Enterprise General Object Storing and Messaging
System" (or some names big-guys at MS, Sun, IBM can introduce). We
love memcache, only.
But, last but not least: I think all ClientAPI version (C Java .NET,
PHP, Perl, ...) MUST understand (get and set and incr/decr) native
data formats, at least 32bit int and string. I think we need it, and
I'll waiting.
Because of our solution has complex parts, several programs in several
programming languages, SO I want they (programs, websites, tools,
bots) talk the same language on the same channel. The common channel
is memcache, and we use native format for values, the common language
is STRING format, only string. All values we use to test are string.
We do the seriallization (simple ini-like, XML or JSON, all encode in
UTF-8) before put it to our MemcacheHelper to write to memcacheD.
@Dustin: I think you should left the seriallization jobs to whom want
to use memcache. They have several systems, they need 'em talk
together, so they know what the most suitable seriallization way to
implement, NOT YOU. They (the wild wide world who use memcache)
develop program (maybe big program) know how to serialize :D , don't
worry. Thanks again.
Memcache must operate well, fast, reliable and simple. Memcache is
useful and famous because of that feature. Add more complex
transparent general enterprise blah blah features will make memcache
become something like "Enterprise General Object Storing and Messaging
System" (or some names big-guys at MS, Sun, IBM can introduce). We
love memcache, only.
But, last but not least: I think all ClientAPI version (C Java .NET,
PHP, Perl, ...) MUST understand (get and set and incr/decr) native
data formats, at least 32bit int and string. I think we need it, and
I'll waiting.
Henrik Schröder
Oct 17, 2008, 8:04:57 AM10/17/08
to memc...@googlegroups.com
Hi Locke,
Memcached itself only handles byte-arrays, with one exception which is the incr and decr commands. Other than that, the server has no real idea about what you stored. Memcached also allows you to store a byte-flag along with your data so that your client can know what is in the byte-array when it gets it. Not all clients do this, some old ones from before the flag existed use the first byte of the byte-array as their flag. Also, there is no convention around what different flag values should mean. There is also the problem that different languages have different sets of native data types that can be serialized, and there is of course the problem that different languages serialize complex objects completely different, and you can't use a C# object in Java and vice versa.
I completely disagree with you that clients should not serialize data, most people are simply not interested in that part, and most people use only one platform, and providing an efficient standard serializer is therefore a good idea. If you use my client (BeITMemcached) you can store byte-arrays, handle the serialization yourself, and my client won't do anything to the data. It might compress it, but you can switch that behaviour off. When I wrote my client, I looked around what flag values other clients were using, and it's all different, so I made my own version. :-)
As a sort of minimum client interoperability, if all clients used the same flag value for byte-arrays, that'd be enough. I'm using flag value 0, I think some other clients do this as well. To solve your immediate problems, take the three clients you have, and modify them to use the same flag value for byte-arrays, and handle all serialization yourself. For BeITMemcached, there's an enum with the flags in Serializer.cs, it's really easy to change.
To the rest of the people on this list, don't you think it'd be a good idea to get some standardization going for the flags with the release of the binary protocol? The actively supported clients will make implementation changes for it, and there will be new clients made for languages that have no actively supported client, so there will be a lot of implementation going on, and if we can standardize the flags, we can get everyone on the same page, which would be pretty nice.
The minimum thing would be a flag for unmodified byte-arrays. If everyone agrees on a flag value for that, we technically get full client interoperability because users can then build custom serialization on top of that.
To make life a bit easier we should have a standard flag value for UTF8-encoded strings, and maybe also define a flag value for string-encoded numbers, the ones that the incr and decr commands work on.
Finally, JSON objects for data structures like arrays and associative arrays of strings and numbers.
I checked the latest version of the Binary protocol spec at http://code.sixapart.com/svn/memcached/branches/binary/server/doc/protocol-binary.txt and it mentions "Data Type (reserved for future use)". Well. How about we define it now? It's one byte (unlike the current flags which is two bytes), so just reserve the upper half for client-specific values, but define a few common ones in the lower half. Is Aaron Stone reading this list? (He's listed as the author of the spec). Is "Sean" reading it? It says in the spec that he will use it soon. :-)
/Henrik Schröder
Memcached itself only handles byte-arrays, with one exception which is the incr and decr commands. Other than that, the server has no real idea about what you stored. Memcached also allows you to store a byte-flag along with your data so that your client can know what is in the byte-array when it gets it. Not all clients do this, some old ones from before the flag existed use the first byte of the byte-array as their flag. Also, there is no convention around what different flag values should mean. There is also the problem that different languages have different sets of native data types that can be serialized, and there is of course the problem that different languages serialize complex objects completely different, and you can't use a C# object in Java and vice versa.
I completely disagree with you that clients should not serialize data, most people are simply not interested in that part, and most people use only one platform, and providing an efficient standard serializer is therefore a good idea. If you use my client (BeITMemcached) you can store byte-arrays, handle the serialization yourself, and my client won't do anything to the data. It might compress it, but you can switch that behaviour off. When I wrote my client, I looked around what flag values other clients were using, and it's all different, so I made my own version. :-)
As a sort of minimum client interoperability, if all clients used the same flag value for byte-arrays, that'd be enough. I'm using flag value 0, I think some other clients do this as well. To solve your immediate problems, take the three clients you have, and modify them to use the same flag value for byte-arrays, and handle all serialization yourself. For BeITMemcached, there's an enum with the flags in Serializer.cs, it's really easy to change.
To the rest of the people on this list, don't you think it'd be a good idea to get some standardization going for the flags with the release of the binary protocol? The actively supported clients will make implementation changes for it, and there will be new clients made for languages that have no actively supported client, so there will be a lot of implementation going on, and if we can standardize the flags, we can get everyone on the same page, which would be pretty nice.
The minimum thing would be a flag for unmodified byte-arrays. If everyone agrees on a flag value for that, we technically get full client interoperability because users can then build custom serialization on top of that.
To make life a bit easier we should have a standard flag value for UTF8-encoded strings, and maybe also define a flag value for string-encoded numbers, the ones that the incr and decr commands work on.
Finally, JSON objects for data structures like arrays and associative arrays of strings and numbers.
I checked the latest version of the Binary protocol spec at http://code.sixapart.com/svn/memcached/branches/binary/server/doc/protocol-binary.txt and it mentions "Data Type (reserved for future use)". Well. How about we define it now? It's one byte (unlike the current flags which is two bytes), so just reserve the upper half for client-specific values, but define a few common ones in the lower half. Is Aaron Stone reading this list? (He's listed as the author of the spec). Is "Sean" reading it? It says in the spec that he will use it soon. :-)
/Henrik Schröder
Dustin
Oct 17, 2008, 11:31:08 AM10/17/08
to memcached
On Oct 17, 1:17 am, LockeVN <lock...@gmail.com> wrote:
> @Dustin: I think you should left the seriallization jobs to whom want
> to use memcache. They have several systems, they need 'em talk
> together, so they know what the most suitable seriallization way to
> implement, NOT YOU. They (the wild wide world who use memcache)
> develop program (maybe big program) know how to serialize :D , don't
> worry. Thanks again.
serialization in any way that makes sense for your application. I
think I've made it quite easy to do so.
Dustin
Oct 17, 2008, 1:06:13 PM10/17/08
to memcached
On Oct 17, 5:04 am, "Henrik Schröder" <skro...@gmail.com> wrote:
> To the rest of the people on this list, don't you think it'd be a good idea
> to get some standardization going for the flags with the release of the
> binary protocol? The actively supported clients will make implementation
> changes for it, and there will be new clients made for languages that have
> no actively supported client, so there will be a lot of implementation going
> on, and if we can standardize the flags, we can get everyone on the same
> page, which would be pretty nice.
> The minimum thing would be a flag for unmodified byte-arrays. If everyone
> agrees on a flag value for that, we technically get full client
> interoperability because users can then build custom serialization on top of
> that.
flexible transcoding mechanisms. If it's just byte arrays, it seems
like that'd put a burden on the users to pack more info into the data
since the flags would be unavailable.
> To make life a bit easier we should have a standard flag value for
> UTF8-encoded strings, and maybe also define a flag value for string-encoded
> numbers, the ones that the incr and decr commands work on.
of flags observed in the wild. I'm not sure if this is the latest:
http://www.hjp.at/zettel/m/memcached_flags.rxml
Convergence would be good.
> Finally, JSON objects for data structures like arrays and associative arrays
> of strings and numbers.
> I checked the latest version of the Binary protocol spec athttp://code.sixapart.com/svn/memcached/branches/binary/server/doc/pro...
> it mentions "Data Type (reserved for future use)". Well. How about we
> define it now?
It still feels like a bad idea, and it's not stored anywhere.
> define it now?
It's in there because there was space and someone really, really
wanted it. The idea is that flags will be available for user
applications, but clients will be able to have their own set of flags
that client users don't get access to. Just sounds rather messy to
me. I'd want to expose them all because I *still* think users should
be able to define their own encoding formats for their own
applications.
Ciaran
Oct 27, 2008, 4:50:23 AM10/27/08
to memc...@googlegroups.com
> I checked the latest version of the Binary protocol spec athttp://code.sixapart.com/svn/memcached/branches/binary/server/doc/pro...
> it mentions "Data Type (reserved for future use)". Well. How about weIt still feels like a bad idea, and it's not stored anywhere.
> define it now?
It's in there because there was space and someone really, really
wanted it. The idea is that flags will be available for user
applications, but clients will be able to have their own set of flags
that client users don't get access to. Just sounds rather messy to
me. I'd want to expose them all because I *still* think users should
be able to define their own encoding formats for their own
applications.
I did try to start pulling together the information (pre-binary protocol) here on the memcached wiki http://www.socialtext.net/memcached/index.cgi?cross_platform_serialisation_support . Unfortunately I ran out of time (I needed this for a work thing) and had to go with a limited implementation locally to me, but I still think there is value in the approach.
- Cj.
Dustin
Oct 27, 2008, 5:23:08 AM10/27/08
to memcached
On Oct 27, 1:50 am, Ciaran <ciar...@gmail.com> wrote:
> I did try to start pulling together the information (pre-binary protocol)
level interoperability issue.
Your work in getting this stuff together is the first step, though.
Reply all
Reply to author
Forward
0 new messages