performance issue: alternate to cjson/msgpack for lua table serialization

[RJoshi]

Hello,

  I am using cjson  library for de/serialization. Recently replaced it with  msgpack which improved performance and reduced the size but still looking for better performance alternatives.

  

  

 NOTE: I did try lua-capnproto but due to packing not supported (https://github.com/calio/lua-capnproto/issues/9) , it increases the size significantly. 

Thanks,

Joshi

[Mathew Heard]

Have you considered luabins?  

[RJoshi]

Thanks Mathew. I will look into luabins. Looking at the issue#1, it has not been validated against LuaJIT but will defiantly give a try. 

[SplitIce]

We havent seen any issue using it with LuaJIT. And its blazingly fast (not that I've done any scientific comparisons with anything other than cjson).

--
You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/TLz-ejLcU9E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[RJoshi]

I am getting error while loading using luajit on mac-osx:  

https://github.com/agladysh/luabins/issues/2

[RJoshi]

Resolved when below command is used:

make LUA_DIR=/opt/openresty/luajit LUA_INCDIR=/opt/openresty/luajit/include/luajit-2.1 LUA_LIBDIR=/opt/openresty/luajit/lib

[RJoshi]

I did some benchmark for a small lua table.  Luabins is twice faster than msgpack but generates the twice the size of serialized data.

 

Lua Table:  local payload = {iat=1468636171,iss="2b0ab90ab1ec40abbde8ffe6859c23e6",gt="CC",exp=1468643371,e="i"}

Iteration: 5M

serialized data size:

  bitser:69

  luabins:115

  msgpack:68

serialize:

   bitser: elapsed time: 15.93

   luabins:elapsed time: 2.98

   msgpack length:elapsed time: 5.94

deserialize:

  bitser: elapsed time: 1.00

  luabins:elapsed time: 0.52

  msgpack:elapsed time: 0.50

[Yichun Zhang (agentzh)]

Hello!

On Tue, Jul 12, 2016 at 4:00 AM, RJoshi wrote:
> I am using cjson library for de/serialization. Recently replaced it with
> msgpack which improved performance and reduced the size but still looking
> for better performance alternatives.
>

Maybe you should consider avoiding serialization/de-serialization on
your hot code paths altogether by caching your Lua data structure
directly with in-VM caches like lua-resty-lrucache. Out-of-VM caches
like lua_shared_dict or even external services like memcached or redis
should only be serve as a secondary cache layer, which can stay out of
the hot code paths.

We did this a lot in our Lua CDN and we no longer bother finding a
faster serialization/de-serialization method since it's barely visible
in our online flame graphs.

Regards,
-agentzh