Twemcache data structures/scripting support

[mgal...@hawk.iit.edu]

Hi,

I'm a student interesting in contributing to twemcache in the gsoc data structures and scripting support project.

It is not totally clear what you mean by data structures support:

Storing an entire data structure,Shouldn't the serialization of data structures is a responsibility of the twemcache clients?

Or do you mean being able to store data structures in the cache and access individual elements of the data structure inside the cache? 

e.g.

memcache_set_list(id, [1,2,3])

memcache_get_list_head(id)

This seems a counterintuitive way of using the cache and presents several challenges because of slabs, i.e the elements in the data structure should all be the same size or it would be scattered in different slab classes and would be harda track, and eviction, an element of the data structure would mean the whole ds needs to be evicted. Otherwise the data structure would be in an inconsistent state.

Any feedback would be greatly appreciated.

Thanks

[Yao Yue]

You can certainly do serialization on the server side and store the whole thing as one object, which is exactly what Redis does.

Please read the documentation/code about slabs- they aren't the unit for storing data.

[maur...@zebstudios.com]

Hi Yao,

Thanks for your reply.

I understand that the items (mc_item) are the actual unit for storing data. I've downloaded and studied the code. What I've ended up doing was create a list storing it sequentially in memory using the items. Storing the total length of the list in the nbyte field of the mc_item struct and the individual size of each element as a 4 byte value before each element. This gives the flexibility to store items of variable size. The structure looks something like this:

   --------------------------------------------------------------------------------------------------------------------------------------------------

   | item_1 size | item_1 payload | item_2 size | item_2 payload |     ...   | item_n size | item_n payload  | 

   ---------------------------------------------------------------------------------------------------------------------------------------------------

The way I store it in cache is somewhat similar to how the append functionality works. I calculate the new size of the list to see if its stille fit in the current slab or I have to move to another slab class. I haven't had much time to investigate the details of this and memory alignment. It was a quick solution that worked to see the inner workings of twemcache this weekend, but need to be polished.

I'm also still debating what could be an appropriate structure for a dictionary. I'm undecided between a balanced tree or storing a hash table at the beginning of the item.

I also modified a little bit a python client to insert values to the list, but I couldn't manage to retrive them from the cache (the connection closes and only sends an END marker. I have to check if this is an issue on the server side or the client side).

For the scripting support I see that redis has a nice interpreter of Lua built in. So I think its worth a shot to see how Redis did it.

I would love to get your thoughts on my strategy.

Thanks,

Mauricio