Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
開源項目 rapidjson
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  Messages 1 - 25 of 26 - Collapse all  -  Translate all to Translated (View all originals)   Newer >
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Milo Yip  
View profile   Translate to Translated (View Original)
 More options Nov 18 2011, 1:40 pm
From: Milo Yip <milo...@gmail.com>
Date: Sat, 19 Nov 2011 02:40:06 +0800
Local: Fri, Nov 18 2011 1:40 pm
Subject: 開源項目 rapidjson
http://code.google.com/p/rapidjson/

一直都沒能成功把一些代碼開源。這次嘗試做點盡量簡單的,工作上又可能用到的,所以就寫了一個JSON庫。

rapidjson是一個高性能的C++ JSON
parser和generator,提供SAX/DOM風格的API,支持UTF8/16/32,可選用SSE2/SSE4.2加速,內存使用緊湊。全個庫只有 1600多行代碼的頭文件。

在性能方面,rapidjson受rapidxml啟發(從名字也能想到吧),支持in-situ
parsing。這是指會把字符串的解碼直接寫到源字符串去。我比較了其他兩個C/C++ JSON parser,這裡只寫一點數據,有空再整理。

在VC2010的一個測試中,parse一個672KB的JSON文件至DOM,rapidjson 730μs,YAJL
6149μs,JsonCpp 6680μs,而strlen()也需要495μs。能夠接近strlen()的速度應該算是不錯,但這和VC2010的strlen()性能較差也有關係。c ygwin中跑gcc的strlen()只需189μs。

性能源於幾方面,in-situ parsing、用SSE4.2跳過whitespace,使用lookup table計算pow10(int
x)等等。另外還有C++的強項,我使用template進行功能的組合,沒有虛函數的多態,也充分利用到inline的好處。

在VC2010中跳過1M個whitespace,strspn()要3074μs,而SSE4.2加速後只需100μs。這段代碼對很多項目都能使用。

另外,原本能選用C++ exception,後來加入setjmp/longjmp後,覺得完全沒需要使用exception,這方面也能提高少量的性能。

預設的內存allocator是只分配不釋放的,而且parsing時特別設計過,不會產生如std::vector::push_back()時的copy及 內存浪費。每個JSON
value在32-bit/64-bit系統中分別佔16及20 bytes(不含字符串)。

設計上提取了一些concept去組合不同功能。例如Reader會從實現了Stream
concept的類讀取字符,之後parse時,把事件傳送給實現了Handler
concept的類。例如PrettyWritter實現了Handler,而產生JSON文本的流,那麼把二者聯合使用:

Reader reader;
FileStream is(stdin);

// Prepare writer and output stream.
FileStream os(stdout);
PrettyWriter<FileStream> writer(os);

// JSON reader parse from the input stream and let writer generate the output.
if (!reader.Parse<0>(is, writer))
        fprintf(stderr, "\nError(%u): %s\n",
(unsigned)reader.GetErrorOffset(), reader.GetParseError());

就能validate及"beautify" JSON 文本。

在rapidjson中,Document是用來實現DOM風格的API。特別之處是它能遍歷自己去產生Handler的事件。所以用以下代碼可以把一個Doc ument聯結PrettyWriter把內容寫成JSON文本的流:

Document document;
document.Parse<0>(json);

// modify document ...

FileStream f(stdout);
PrettyWriter<FileStream> writer(f);
document.Accept(writer);        // Accept() traverses the DOM and generates
Handler events.

小弟第一次做開源項目,希望大家能用得著,也多給意見。

--
Milo Yip

http://www.cnblogs.com/miloyip/
http://weibo.com/miloyip/
http://twitter.com/miloyip/


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
翁翊成  
View profile   Translate to Translated (View Original)
 More options Nov 18 2011, 10:40 pm
From: 翁翊成 <weng...@gmail.com>
Date: Sat, 19 Nov 2011 11:40:15 +0800
Local: Fri, Nov 18 2011 10:40 pm
Subject: Re: [TL] 開源項目 rapidjson

开源项目现在更流行使用github,方便其他感兴趣的人参与并贡献代码。

2011/11/19 Milo Yip <milo...@gmail.com>

--
翁翊成
http://weibo.com/wengych

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
马博文  
View profile   Translate to Translated (View Original)
 More options Nov 18 2011, 11:58 pm
From: 马博文 <iambowe...@gmail.com>
Date: Sat, 19 Nov 2011 12:58:13 +0800
Local: Fri, Nov 18 2011 11:58 pm
Subject: Re: [TL] 開源項目 rapidjson

应该附上连接啊,www.github.com

在 2011年11月19日 上午11:40,翁翊成 <weng...@gmail.com>写道:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
shiwei xu  
View profile   Translate to Translated (View Original)
 More options Nov 19 2011, 10:12 am
From: shiwei xu <xushiwe...@gmail.com>
Date: Sat, 19 Nov 2011 23:12:55 +0800
Local: Sat, Nov 19 2011 10:12 am
Subject: Re: [TL] 開源項目 rapidjson

非常实用的一个库。:)
我个人也推荐用 github.com 作为 hosting,比 google code 有更好的社区支持,比如他人可以请求合并修改的代码到主干。

2011/11/19 Milo Yip <milo...@gmail.com>

--
许式伟
https://qbox.me

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Milo Yip  
View profile   Translate to Translated (View Original)
 More options Nov 19 2011, 10:24 am
From: Milo Yip <milo...@gmail.com>
Date: Sat, 19 Nov 2011 23:24:48 +0800
Local: Sat, Nov 19 2011 10:24 am
Subject: Re: 開源項目 rapidjson
今天寫了一篇rapidjson的性能比較,及分析背後原因的文檔

http://code.google.com/p/rapidjson/wiki/Performance

至於github,鑑於本人不懂Git,可能留待他日再作轉移。

我有空會繼續寫使用手冊、內存比較、設計等文檔。

希望各位多給意見。

--
Milo Yip

http://www.cnblogs.com/miloyip/
http://weibo.com/miloyip/
http://twitter.com/miloyip/


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Bill Hsu  
View profile   Translate to Translated (View Original)
 More options Nov 19 2011, 11:30 am
From: Bill Hsu <billhs...@gmail.com>
Date: Sun, 20 Nov 2011 00:30:25 +0800
Local: Sat, Nov 19 2011 11:30 am
Subject: Re: [TL] Re: 開源項目 rapidjson

谢谢Milo分享!
考完期末仔细研究一下。

Best,
Bill Hsu

Blog: http://BillHsu.me
Twitter: http://twitter.com/1991bill
Weibo: http://weibo.com/billhsu

2011/11/19 Milo Yip <milo...@gmail.com>


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
HaoPeiQiang  
View profile   Translate to Translated (View Original)
 More options Nov 19 2011, 9:05 pm
From: HaoPeiQiang <HaoPeiQi...@gmail.com>
Date: Sun, 20 Nov 2011 10:05:01 +0800
Local: Sat, Nov 19 2011 9:05 pm
Subject: Re: [TL] Re: 開源項目 rapidjson

很赞,虽然我的项目里面json解析效率不重要

2011/11/20 Bill Hsu <billhs...@gmail.com>

--
Tinyfool的Blog http://tiny4.org/blog/
Tiny4Cocoa http://tiny4.org/cocoa/
myTwitter: http://twitter.com/tinyfool

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
朱金灿  
View profile   Translate to Translated (View Original)
 More options Nov 26 2011, 6:05 am
From: 朱金灿 <clever...@gmail.com>
Date: Sat, 26 Nov 2011 19:05:54 +0800
Local: Sat, Nov 26 2011 6:05 am
Subject: Re: [TL] Re: 開源項目 rapidjson

json主要用于网络传输吧,如果作为配置文件,它比起xml文件的优势是什么?

在 2011年11月20日 上午10:05,HaoPeiQiang <HaoPeiQi...@gmail.com>写道:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
翁翊成  
View profile   Translate to Translated (View Original)
 More options Nov 26 2011, 10:25 am
From: 翁翊成 <weng...@gmail.com>
Date: Sat, 26 Nov 2011 23:25:39 +0800
Local: Sat, Nov 26 2011 10:25 am
Subject: Re: [TL] Re: 開源項目 rapidjson

没见过用json当配置文件的。

2011/11/26 朱金灿 <clever...@gmail.com>

--
翁翊成
http://weibo.com/wengych

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
windstorm  
View profile   Translate to Translated (View Original)
 More options Nov 26 2011, 10:31 am
From: windstorm <likunarmstr...@gmail.com>
Date: Sat, 26 Nov 2011 08:31:05 -0700
Local: Sat, Nov 26 2011 10:31 am
Subject: Re: [TL] Re: 開源項目 rapidjson
用json当配置文件的多了去了。

配置文件的特点不外乎这几点,轻便简单易读,能跨平台跨语言,支持unicode编码。json是唯一完美支持这几个特性的。

--------------------------------------------------------------------------- -------
Yours Sincerely
Kun

gplus.to/kunli

2011/11/26 翁翊成 <weng...@gmail.com>:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tux9  
View profile   Translate to Translated (View Original)
 More options Nov 26 2011, 6:03 pm
From: Tux9 <tuxd...@gmail.com>
Date: Sun, 27 Nov 2011 00:03:40 +0100
Local: Sat, Nov 26 2011 6:03 pm
Subject: Re: [TL] Re: 開源項目 rapidjson

json一点都不简单易读

那你说说xml和直接A=b这两种配置文件,相比缺点是什么

2011/11/26 windstorm <likunarmstr...@gmail.com>


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
qiaojie  
View profile   Translate to Translated (View Original)
 More options Nov 27 2011, 12:51 am
From: qiaojie <qiao...@gmail.com>
Date: Sun, 27 Nov 2011 13:51:33 +0800
Local: Sun, Nov 27 2011 12:51 am
Subject: Re: [TL] Re: 開源項目 rapidjson

json比xml简洁,解析起来也简单,用来做配置文件或者序列化可阅读的对象都是很好的选择。

2011/11/26 翁翊成 <weng...@gmail.com>


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tux9  
View profile   Translate to Translated (View Original)
 More options Nov 27 2011, 7:02 am
From: Tux9 <tuxd...@gmail.com>
Date: Sun, 27 Nov 2011 13:02:35 +0100
Local: Sun, Nov 27 2011 7:02 am
Subject: Re: [TL] Re: 開源項目 rapidjson

大部分配置文件都不会频繁读写,所以简洁没用,解析简单也用处不大(反正有现成的包),每次读写多点CPU周期和磁盘读写跟本不影响那可以忽略的执行时间。人肉 读写简单才是王道

2011/11/27 qiaojie <qiao...@gmail.com>


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
zhang3  
View profile   Translate to Translated (View Original)
 More options Nov 27 2011, 7:07 am
From: zhang3 <openau...@gmail.com>
Date: Sun, 27 Nov 2011 20:07:50 +0800
Local: Sun, Nov 27 2011 7:07 am
Subject: Re: [TL] Re: 開源項目 rapidjson
还是用小型的函数式脚本语言做配置更能满足各种复杂的要求。

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Lich_Ray  
View profile   Translate to Translated (View Original)
 More options Nov 27 2011, 7:15 am
From: Lich_Ray <solo.l...@gmail.com>
Date: Sun, 27 Nov 2011 06:15:35 -0600
Local: Sun, Nov 27 2011 7:15 am
Subject: Re: [TL] Re: 開源項目 rapidjson

On Sun, Nov 27, 2011 at 6:07 AM, zhang3 <openau...@gmail.com> wrote:
> 还是用小型的函数式脚本语言做配置更能满足各种复杂的要求。

其中 lisp 风格的十分好用。如果是用 guile 一类扩展的话还能嵌入一些逻辑。

--
Zhihao Yuan, nickname Lich_Ray
God is in his heaven, all's right with the world.
-------------------------------------------------
let focus = 'computing' in where:
http://lichray.javaeye.com
let focus = 'computing' in here:
http://let-in.blogspot.com


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
missdeer  
View profile   Translate to Translated (View Original)
 More options Nov 28 2011, 1:52 am
From: missdeer <missd...@gmail.com>
Date: Sun, 27 Nov 2011 22:52:26 -0800 (PST)
Local: Mon, Nov 28 2011 1:52 am
Subject: Re: 開源項目 rapidjson
Mark~这个有意思,晚上回去看看能不能合到自己的程序中去~~

On Nov 19, 2:40 am, Milo Yip <milo...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Zhangming Niu  
View profile   Translate to Translated (View Original)
 More options Nov 28 2011, 9:43 am
From: Zhangming Niu <niuzhangm...@gmail.com>
Date: Mon, 28 Nov 2011 15:43:59 +0100
Local: Mon, Nov 28 2011 9:43 am
Subject: Re: [TL] Re: 開源項目 rapidjson

宝马比起奔驰的优势是什么。。。

2011/11/28 missdeer <missd...@gmail.com>

--
--------------------------------------------------------------------
Best Regards,

Zhangming Niu


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
missdeer  
View profile   Translate to Translated (View Original)
 More options Nov 28 2011, 8:08 pm
From: missdeer <missd...@gmail.com>
Date: Mon, 28 Nov 2011 17:08:53 -0800 (PST)
Local: Mon, Nov 28 2011 8:08 pm
Subject: Re: 開源項目 rapidjson
没有两种车都开过一段时间,谁能说出到底各有什么优缺点。。。

On Nov 28, 10:43 pm, Zhangming Niu <niuzhangm...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
johnnie chan  
View profile   Translate to Translated (View Original)
 More options Nov 28 2011, 8:24 pm
From: johnnie chan <ich...@gmail.com>
Date: Mon, 28 Nov 2011 17:24:25 -0800 (PST)
Local: Mon, Nov 28 2011 8:24 pm
Subject: Re: 開源項目 rapidjson
似乎yajl的作者对Milo的这个performance比较很不乐意......
https://github.com/lloyd/yajl_vs_rapidjson

On Nov 19, 11:24 pm, Milo Yip <milo...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Milo Yip  
View profile   Translate to Translated (View Original)
 More options Nov 28 2011, 9:03 pm
From: Milo Yip <milo...@gmail.com>
Date: Tue, 29 Nov 2011 10:03:13 +0800
Local: Mon, Nov 28 2011 9:03 pm
Subject: Re: [TL] Re: 開源項目 rapidjson
這個一星期前和他溝通過了。之前的問題在於例子中用fgetc()比fread()慢很多。在我的比較中,只使用了內存中的字符串去測量parsing/gen eration速度。我會待完成一些新功能後,發佈0.2版本時再更新性能測試。

以下是我回覆YAJL作者的信:

Today I have investigated the issue. And have found that the
inefficiency is due to fgetc() and fputc() for stream read/write, and
perftest was only testing in-memory parsing.

This is the 0.1 version I reproduced on Ubuntu 64-bit:

$ time pretty_release_x64_gmake.exe < test_file.json > /dev/null

real    0m0.180s
user    0m0.160s
sys     0m0.010s

$ time ./json_reformat < test_file.json > /dev/null

real    0m0.118s
user    0m0.100s
sys     0m0.000s

It is very similar to the statistics you have reproduced.

Without chaning the code of the parser/generator in rapidxml, I added
two stream classes which use fread()/fwrite(). And each stream
allocates 64KB buffer in pretty (since json_reformat uses 64KB buffer
for reading). Then:

$ time pretty_release_x64_gmake.exe < test_file.json > /dev/null

real    0m0.052s
user    0m0.040s
sys     0m0.000s

Another comparison with actual write:

$ time json_reformat < test_file.json > a

real    0m0.121s
user    0m0.080s
sys     0m0.020s

$ time pretty_release_x64_gmake.exe < test_file.json > a

real    0m0.070s
user    0m0.050s
sys     0m0.010s

Finally, I think in these reformatting tests, rapidjson can be further
optimized with just validating JSON string/numbers but not decoding
and re-encoding them.

2011/11/29 johnnie chan <ich...@gmail.com>:

--
Milo Yip

http://www.cnblogs.com/miloyip/
http://weibo.com/miloyip/
http://twitter.com/miloyip/


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
rockeet febird  
View profile   Translate to Translated (View Original)
 More options Nov 29 2011, 5:36 am
From: rockeet febird <rock...@gmail.com>
Date: Tue, 29 Nov 2011 02:36:08 -0800 (PST)
Local: Tues, Nov 29 2011 5:36 am
Subject: Re: 開源項目 rapidjson
精神可嘉!
我发现你的 FindMember 使用的是 sequencial find, member 少的时候无所谓, member 多的话就悲剧了,
时间复杂度是 O(n).

On Nov 19, 2:40 am, Milo Yip <milo...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Milo Yip  
View profile   Translate to Translated (View Original)
 More options Nov 29 2011, 11:17 pm
From: Milo Yip <milo...@gmail.com>
Date: Wed, 30 Nov 2011 12:17:36 +0800
Local: Tues, Nov 29 2011 11:17 pm
Subject: Re: [TL] Re: 開源項目 rapidjson
是的,打算之後加入簡單的hashing。不過這樣會改變原來的member次序。

2011/11/29 rockeet febird <rock...@gmail.com>:

> 精神可嘉!
> 我发现你的 FindMember 使用的是 sequencial find, member 少的时候无所谓, member 多的话就悲剧了,
> 时间复杂度是 O(n).

--
Milo Yip

http://www.cnblogs.com/miloyip/
http://weibo.com/miloyip/
http://twitter.com/miloyip/


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
jiang yu  
View profile   Translate to Translated (View Original)
 More options Nov 30 2011, 1:23 am
From: jiang yu <yu.jiang....@gmail.com>
Date: Wed, 30 Nov 2011 14:23:56 +0800
Local: Wed, Nov 30 2011 1:23 am
Subject: Re: [TL] 開源項目 rapidjson
跟strlen保持在一个量级确实难以置信!

在 2011年11月19日 上午2:40,Milo Yip <milo...@gmail.com> 写道:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
jiang yu  
View profile  
 More options Nov 30 2011, 1:32 am
From: jiang yu <yu.jiang....@gmail.com>
Date: Wed, 30 Nov 2011 14:32:42 +0800
Local: Wed, Nov 30 2011 1:32 am
Subject: Re: [TL] 開源項目 rapidjson
compile test.sln in vs2010:

1>c:\users\mos\svn\rapidjson-read-only\include\rapidjson\rapidjson.h(85):
fatal error C1083: Cannot open include file: 'allocators.h': No such
file or directory

在 2011年11月30日 下午2:23,jiang yu <yu.jiang....@gmail.com> 写道:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
rockeet febird  
View profile   Translate to Translated (View Original)
 More options Nov 30 2011, 4:58 am
From: rockeet febird <rock...@gmail.com>
Date: Wed, 30 Nov 2011 01:58:54 -0800 (PST)
Local: Wed, Nov 30 2011 4:58 am
Subject: Re: 開源項目 rapidjson
可以用:
http://blog.csdn.net/whinah/article/details/6819050
http://blog.csdn.net/whinah/article/details/6876182

On Nov 30, 12:17 pm, Milo Yip <milo...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Messages 1 - 25 of 26   Newer >
« Back to Discussions « Newer topic     Older topic »