除了cPickle,cjson外还有没有更高效点的序列化库了？
我欲乘风去
除了这俩还有啥可用高效的序列化库了？
jyf
要不你试试BSON?
On Tue, Jul 12, 2011 at 04:35:47AM -0700, 我欲乘风去 wrote:
> 项目里用redis作消息队列，测下来大部份时间消耗在了cPickle上。试了试cjson，效率也高不了多少。
> 除了这俩还有啥可用高效的序列化库了？
>
> --
> 来自: python-cn`CPyUG`华蟒用户组(中文Python技术邮件列表)
> 发言: pyth...@googlegroups.com
> 退订: python-cn+...@googlegroups.com (向此发空信即退!)
> 详情: http://code.google.com/p/cpyug/wiki/PythonCn
> 严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
> 强烈: 建议使用技巧: 如何有效地报告Bug
> http://www.chiark.greenend.org.uk/%7Esgtatham/bugs-cn.html
--
cheers
Yunfan Jiang
{'nick':['jyf', 'geek42'], 'im': {'gtalk': 'jyf...@gmail.com', 'irc': 'irc.freenode.net#ubuntu-cn'}, 'blog': 'http://geek42.info', 'interesting': {'teck': ['linux', 'python', 'lua', 'c', 'nosql', 'redis', 'nginx'], 'history': ['chinese history',], 'sf': [42,], 'music': ['NewAge style', 'chinese old theme', 'Any strange music']}}
simomo
在 Tue, 12 Jul 2011 19:37:14 +0800，jyf <jyf...@gmail.com> 写道:
> 想说 Protobuf 不过不灵活
> 要不你试试BSON?
>
>
> On Tue, Jul 12, 2011 at 04:35:47AM -0700, 我欲乘风去 wrote:
>> 项目里用redis作消息队列，测下来大部份时间消耗在了cPickle上。试了试cjson，效率也高不了多少。
>> 除了这俩还有啥可用高效的序列化库了？
>>
>> --
>> 来自: python-cn`CPyUG`华蟒用户组(中文Python技术邮件列表)
>> 发言: pyth...@googlegroups.com
>> 退订: python-cn+...@googlegroups.com (向此发空信即退!)
>> 详情: http://code.google.com/p/cpyug/wiki/PythonCn
>> 严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
>> 强烈: 建议使用技巧: 如何有效地报告Bug
>> http://www.chiark.greenend.org.uk/%7Esgtatham/bugs-cn.html
>
--
我欲乘风去
已有答案了，tnetstring不错，先用几天看有没有问题。。速度居然是cPickle的5倍
On Jul 12, 7:37 pm, jyf <jyf1...@gmail.com> wrote:
> 想说 Protobuf 不过不灵活
> 要不你试试BSON?
>
> On Tue, Jul 12, 2011 at 04:35:47AM -0700, 我欲乘风去 wrote:
> > 项目里用redis作消息队列，测下来大部份时间消耗在了cPickle上。试了试cjson，效率也高不了多少。
> > 除了这俩还有啥可用高效的序列化库了？
>
> > --
> > 来自: python-cn`CPyUG`华蟒用户组(中文Python技术邮件列表)
> > 发言: pyth...@googlegroups.com
> > 退订: python-cn+...@googlegroups.com (向此发空信即退!)
> > 详情:http://code.google.com/p/cpyug/wiki/PythonCn
> > 严正: 理解列表! 智慧提问!http://wiki.woodpecker.org.cn/moin/AskForHelp
> > 强烈: 建议使用技巧: 如何有效地报告Bug
> > http://www.chiark.greenend.org.uk/%7Esgtatham/bugs-cn.html
>
> --
> cheers
> Yunfan Jiang
> {'nick':['jyf', 'geek42'], 'im': {'gtalk': 'jyf1...@gmail.com', 'irc': 'irc.freenode.net#ubuntu-cn'}, 'blog': 'http://geek42.info', 'interesting': {'teck': ['linux', 'python', 'lua', 'c', 'nosql', 'redis', 'nginx'], 'history': ['chinese history',], 'sf': [42,], 'music': ['NewAge style', 'chinese old theme', 'Any strange music']}}
imdiot
On 2011��07��12�� 19:35, ����˷�ȥ wrote:
> ��Ŀ����redis����Ϣ���У���������ʱ���������cPickle�ϡ�������cjson��Ч��Ҳ�߲��˶��١�
> ������������ɶ���ø�Ч�����л����ˣ�
>
David.liu
On 2011年07月12日 19:35, 我欲乘风去 wrote:
> 项目里用redis作消息队列，测下来大部份时间消耗在了cPickle上。试了试cjson，效率也高不了多少。
> 除了这俩还有啥可用高效的序列化库了？
>
--
来自: python-cn`CPyUG`华蟒用户组(中文Python技术邮件列表)
发言: pyth...@googlegroups.com
退订: python-cn+...@googlegroups.com (向此发空信即退!)
详情: http://code.google.com/p/cpyug/wiki/PythonCn
严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
--
blog: http://blog.xgarden.net
pansz
楼主好样的，我一直为 cPickle 的效率问题头疼。
Kula
--
来自: python-cn`CPyUG`华蟒用户组(中文Python技术邮件列表)
发言: pyth...@googlegroups.com
退订: python-cn+...@googlegroups.com (向此发空信即退!)
详情: http://code.google.com/p/cpyug/wiki/PythonCn
严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
我欲乘风去
On Jul 12, 10:18 pm, "David.liu" <jxppp....@gmail.com> wrote:
> 试一下 marshal
> 在 2011年7月12日 下午7:56，imdiot <imdio...@gmail.com>写道：
>
> > msgpack?
> > On 2011年07月12日 19:35, 我欲乘风去 wrote:
> > > 项目里用redis作消息队列，测下来大部份时间消耗在了cPickle上。试了试cjson，效率也高不了多少。
> > > 除了这俩还有啥可用高效的序列化库了？
>
> > --
> > 来自: python-cn`CPyUG`华蟒用户组(中文Python技术邮件列表)
> > 发言: pyth...@googlegroups.com
> > 退订: python-cn+...@googlegroups.com (向此发空信即退!)
> > 详情:http://code.google.com/p/cpyug/wiki/PythonCn
> > 严正: 理解列表! 智慧提问!http://wiki.woodpecker.org.cn/moin/AskForHelp
pyleaf
严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
--
Believing is Everything.
limodou
> tnetstring 的确不是一般的快，面对结构化的数据，真的很强悍
>
有时间试试。
--
I like python!
UliPad <<The Python Editor>>: http://code.google.com/p/ulipad/
UliWeb <<simple web framework>>: http://code.google.com/p/uliweb/
My Blog: http://hi.baidu.com/limodou
MuSheng Chen
我欲乘风去
msgpack最快，而且是跨语言的,二进制,但只能打包简单的list,dict,int,string,unicode，在memcache之类的
字符串协议里会有问题。
marshal其次，也是二进制的，可以打包大多数python对象，缺点同msgpack，另文档上说各版本的python实现会不一样，但我这测下
来三台机器分别是stackless 2.6.5 2.7.2 python 2.6.2之间都没出现兼容问题
tnetstring速度第三，字符打包，可以完美支持各种协议的后端，但有一个致命的缺点就是支持类型太少，比如下面的代码mongo里出来的
unicode就无法打包。
以上三者速度差距很小很小。
bson,cPickle,json都很慢。。直接pass
cPickle_test used time 1.32382702827s
marshal_test used time 0.29944396019s
msgpackb_test used time 0.23744893074s
#!/usr/bin/env Python
#coding=utf-8
'''
Created on Jul 13, 2011
@author: apple
'''
import time,functools
import cPickle,marshal,tnetstring,bson,msgpack
import pymongo
data = pymongo.Connection().freeasker.question.find_one()
data.pop('_id')
def timeit(func):
@functools.wraps(func)
def __do__(*args,**wk):
start = time.time()
result= func(*args,**wk)
print '%s used time %ss'%(func.__name__,time.time()-start)
return result
return __do__
@timeit
def cPickle_test(c,data):
for i in xrange(c):
s = cPickle.dumps(data)
cPickle.loads(s)
@timeit
def marshal_test(c,data):
for i in xrange(c):
s = marshal.dumps(data)
marshal.loads(s)
@timeit
def tnetstring_test(c,data):
for i in xrange(c):
s = tnetstring.dumps(data)
tnetstring.loads(s)
@timeit
def bson_test(c,data):
for i in xrange(c):
s = bson.dumps(data)
bson.loads(s)
@timeit
def msgpackb_test(c,data):
# packer = msgpack.Packer()
for i in xrange(c):
s = msgpack.packb(data)
msgpack.unpackb(s)
@timeit
def msgpack_test(c,data):
packer = msgpack.Packer()
for i in xrange(c):
s = packer.pack(data)
packer.unpack(s)
if __name__ == '__main__':
count = 10000
data = pymongo.Connection().freeasker.question.find_one()
data.pop('_id')
cPickle_test(count, data)
marshal_test(count, data)
# tnetstring_test(count, data)
# bson_test(count, data)
# msgpack_test(count, data)
msgpackb_test(count, data)
On Jul 13, 1:20 pm, MuSheng Chen <sheng.2...@gmail.com> wrote:
> 可以試下msgpack
>
> 2011/7/13 limodou <limo...@gmail.com>
>
>
>
>
>
>
>
> > 2011/7/13 pyleaf <lg.feixi...@gmail.com>:
> > > tnetstring 的确不是一般的快，面对结构化的数据，真的很强悍
>
> > 有时间试试。
>
> > --
> > I like python!
> > UliPad <<The Python Editor>>:http://code.google.com/p/ulipad/
> > UliWeb <<simple web framework>>:http://code.google.com/p/uliweb/
> > My Blog:http://hi.baidu.com/limodou
>
> > --
> > 来自: python-cn`CPyUG`华蟒用户组(中文Python技术邮件列表)
> > 发言: pyth...@googlegroups.com
> > 退订: python-cn+...@googlegroups.com (向此发空信即退!)
> > 详情:http://code.google.com/p/cpyug/wiki/PythonCn
> > 严正: 理解列表! 智慧提问!http://wiki.woodpecker.org.cn/moin/AskForHelp
MuSheng Chen
严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
Shell Xu
严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
--
无能者无所求，饱食而遨游，泛若不系之舟
blog: http://shell909090.com/blog/
twitter: http://twitter.com/shell909090
Shell Xu
import binascii
eval(compile(binascii.a2b_base64('aW1wb3J0IGpzb24sIHRpbWVpdCwgY1BpY2tsZSwgbWFyc2hhbApkZWYgcGlja2xlX2R1bXAoZCk6IHJldHVybiBjUGlja2xlLmR1bXBzKGQpCmRlZiB0aW1lX2Z1bmMobG9hZG5hbWUsIGR1bXBuYW1lKTogcmV0dXJuIHRpbWVpdC5UaW1lcigiJXMoJXMoezE6IFsnYScsICdiJ10sICdjJzogKFRydWUsIEZhbHNlKX0pKSIgJSAobG9hZG5hbWUsIGR1bXBuYW1lKSwgJ2Zyb20gX19tYWluX18gaW1wb3J0IConKS50aW1laXQoMTAwMDApCnByaW50ICdqc29uICVmLicgJSB0aW1lX2Z1bmMoJ2pzb24ubG9hZHMnLCAnanNvbi5kdW1wcycpCnByaW50ICdwaWNrbGUgJWYuJyAlIHRpbWVfZnVuYygnY1BpY2tsZS5sb2FkcycsICdwaWNrbGVfZHVtcCcpCnByaW50ICdtYXJzaGFsICVmLicgJSB0aW1lX2Z1bmMoJ21hcnNoYWwubG9hZHMnLCAnbWFyc2hhbC5kdW1wcycpCg=='),'','exec'))
shell@shell-deb:~/workspace/temp/test_py$ python testser.py
<timeit-src>:2: SyntaxWarning: import * only allowed at module level
json 1.008235.
<timeit-src>:2: SyntaxWarning: import * only allowed at module level
pickle 0.081648.
<timeit-src>:2: SyntaxWarning: import * only allowed at module level
marshal 0.020381.
没装tnetstring，所以就没测试。cPickle用protocol 2效率大概上升40%，不过还是没有marshal快。
严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
leopay
Leo Jay
> 我也很喜欢marshal, 谁用谁知道啊
>
http://docs.python.org/library/marshal.html
This is not a general “persistence” module. For general persistence
and transfer of Python objects through RPC calls, see the modules
pickle and shelve. The marshal module exists mainly to support reading
and writing the “pseudo-compiled” code for Python modules of .pyc
files. Therefore, the Python maintainers reserve the right to modify
the marshal format in backward incompatible ways should the need
arise. If you’re serializing and de-serializing Python objects, use
the pickle module instead – the performance is comparable, version
independence is guaranteed, and pickle supports a substantially wider
range of objects than marshal.
--
Best Regards,
Leo Jay
learno
在 2011年7月13日 下午6:04，Leo Jay <python...@gmail.com> 写道：
> 2011/7/13 leopay <leo...@gmail.com>:
>> 我也很喜欢marshal, 谁用谁知道啊
>>
>
> http://docs.python.org/library/marshal.html
>
> This is not a general "persistence" module. For general persistence
> and transfer of Python objects through RPC calls, see the modules
> pickle and shelve. The marshal module exists mainly to support reading
> and writing the "pseudo-compiled" code for Python modules of .pyc
> files. Therefore, the Python maintainers reserve the right to modify
> the marshal format in backward incompatible ways should the need
> arise. If you're serializing and de-serializing Python objects, use
> the pickle module instead - the performance is comparable, version
> independence is guaranteed, and pickle supports a substantially wider
> range of objects than marshal.
>
> --
> Best Regards,
> Leo Jay
>
我欲乘风去
cPickle_test used time 0.863438844681s
marshal_test used time 0.296789169312s
msgpackb_test used time 0.237455844879s
>
>
>
>
>
>
>
> > 2011/7/13 leopay <leo...@gmail.com>:
> >> 我也很喜欢marshal, 谁用谁知道啊
>
> >http://docs.python.org/library/marshal.html
>
> > This is not a general "persistence" module. For general persistence
> > and transfer of Python objects through RPC calls, see the modules
> > pickle and shelve. The marshal module exists mainly to support reading
> > and writing the "pseudo-compiled" code for Python modules of .pyc
> > files. Therefore, the Python maintainers reserve the right to modify
> > the marshal format in backward incompatible ways should the need
> > arise. If you're serializing and de-serializing Python objects, use
> > the pickle module instead - the performance is comparable, version
> > independence is guaranteed, and pickle supports a substantially wider
> > range of objects than marshal.
>
> > --
> > Best Regards,
> > Leo Jay
>
> > --
> > 来自: python-cn`CPyUG`华蟒用户组(中文Python技术邮件列表)
> > 发言: pyth...@googlegroups.com
> > 退订: python-cn+...@googlegroups.com (向此发空信即退!)
> > 详情:http://code.google.com/p/cpyug/wiki/PythonCn
learno
In [24]: %timeit cPickle.loads(cPickle.dumps(data, 2))
100 loops, best of 3: 10.4 ms per loop
In [25]: %timeit msgpack.loads(msgpack.dumps(data))
100 loops, best of 3: 19.9 ms per loop
In [26]: %timeit marshal.loads(marshal.dumps(data))
100 loops, best of 3: 6.97 ms per loop
我不认为速度差异很大
> 严正: 理解列表! 智慧提问! http://wiki.woodpecker.org.cn/moin/AskForHelp
pansz
> 说pickle慢的同学你们能不能先用上protocol这个参数
>
我用过，结果是对我的数据（一个26M的多层dict），需要10秒左右的这个级别才能够 load
但是其他的大约是 2 秒左右这个级别。
pickle/cpickle 对大数据的性能真的不堪用。