[Development] Json support for QtCore
lars....@nokia.com
there were quite a few discussions and requests over the last months to
add Json support as part of QtCore, and maybe integrating one of the
existing parsers.
I was however never really happy with them. Mainly because they all parse
into a QVariant(Map), and don't offer any real API to create and modify
Json. In addition, I believed that it was possible to create a faster
parser, and wanted to have a binary representation for the parsed Json.
So I sat down over the last weeks and implemented hacked on something on
my own. You can see the current state in our first playground project that
Sergio created for me yesterday. You can clone it from
ssh://user...@codereview.qt-project.org:29418/playground/qtbinaryjson.git
Things look rather promising, and I consider the code to be mostly feature
complete. There's work to be done on the API, and documentation is still
completely missing, but most of the other pieces are there.
What I have right now:
* A binary representation of Json, that can be simply mmap'ed from a file,
or copied around. This gives us an extremely cheap way to load and save
Json objects in binary form.
The binary representation is optimized for fast reading. The binary format
is relatively compact and stores the whole object in one blob.
* An API consisting of JsonObject, JsonArray and JsonValue that allows for
fast and easy iteration and modification of the binary format.
The Api still needs some work, and I have to write docs about it.
* Conversion between the binary representation and a QVariantMap
* Conversion to and from text based Json.
The Json parser is a handwritten, recursive parser. It converts Json (utf8
encoded) into the binary representation. I've tried to make it as fast as
possible and let it do a minimum amount of allocations.
I've benchmarked parsing speed and compared to the qjson
(http://qjson.sourceforge.net/). It's around 7 times faster for the
test.json file included in the project (the test case used in qjson as
well).
* It's compact and small. The whole implementation compiles into a 77k
shared library at the moment, with very few relocations.
There's a bit of work still required to finish it, but I would like to
start the discussion about including it into QtCore anyways. I'd also be
happy about any feedback.
Cheers,
Lars
_______________________________________________
Development mailing list
Devel...@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Robin Burchell
On Sat, Dec 10, 2011 at 10:32 PM, <lars....@nokia.com> wrote:
> I've benchmarked parsing speed and compared to the qjson
> (http://qjson.sourceforge.net/). It's around 7 times faster for the
> test.json file included in the project (the test case used in qjson as
> well).
I'd be interested to see how your parser fares compared to girish's
work (existing benchmark results, somewhat outdated probably, found at
http://git.forwardbias.in/?p=qjsonparser.git;a=commit;h=180f3a1ea85adbea4d77e50ff3be3e75af918252)
- those results at least look a lot better than 7 times faster, but
they are also probably different test data.
But, parser aside: in general terms, I do think I agree that
converting to/from QVariant as an "API" isn' all that nice, so your
approach is a nice alternative.
I do think I'd like to see QDataStream operators on QJsonDocument,
btw. sending json documents in binary form to/from files or sockets
would be useful.
Sivan Greenberg
On Sat, Dec 10, 2011 at 11:32 PM, <lars....@nokia.com> wrote:
> there were quite a few discussions and requests over the last months to
> add Json support as part of QtCore, and maybe integrating one of the
> existing parsers.
There were also discussions and even a couple of bug report about
adding "native" (a'la making Json first class citizen) support to QML
as well (I guess this means adding it to the QDeclerativeContext
engine?) , in favor of having one implementation that'd be standard
and fast - https://bugreports.qt.nokia.com/browse/QTBUG-12117 .
>
> I was however never really happy with them. Mainly because they all parse
> into a QVariant(Map), and don't offer any real API to create and modify
> Json. In addition, I believed that it was possible to create a faster
> parser, and wanted to have a binary representation for the parsed Json.
>
I wonder if the QML implementation in the bug report would suffer from
the same (or perhaps worse?) performance issues?
> So I sat down over the last weeks and implemented hacked on something on
> my own. You can see the current state in our first playground project that
> Sergio created for me yesterday. You can clone it from
> ssh://user...@codereview.qt-project.org:29418/playground/qtbinaryjson.git
>
So, could be bind this to have an QML "native" JSON implementation and
handling? Perhaps also to tie it in to have a JsonModel just as the
XmlListModel uses XML?
> Things look rather promising, and I consider the code to be mostly feature
> complete. There's work to be done on the API, and documentation is still
> completely missing, but most of the other pieces are there.
>
/me keen to find a spare weekend to check this out.
> What I have right now:
>
> * A binary representation of Json, that can be simply mmap'ed from a file,
> or copied around. This gives us an extremely cheap way to load and save
> Json objects in binary form.
Would this enable us to store complete binary chunks (files, network
dumps etc..) as BSON allows? I have some use cases for this if this is
possible.
>
> The binary representation is optimized for fast reading. The binary format
> is relatively compact and stores the whole object in one blob.
Is this a standard binary format of JSON, or something of your own devise?
> There's a bit of work still required to finish it, but I would like to
> start the discussion about including it into QtCore anyways. I'd also be
> happy about any feedback.
Many thanks for this!
-Sivan
Ben Lau
http://www.qtcentre.org/threads/46315-QtMapStream-serialization-of-object-class-to-QVariantMap
lars....@nokia.com
>hi,
>
>On Sat, Dec 10, 2011 at 10:32 PM, <lars....@nokia.com> wrote:
>> I've benchmarked parsing speed and compared to the qjson
>> (http://qjson.sourceforge.net/). It's around 7 times faster for the
>> test.json file included in the project (the test case used in qjson as
>> well).
>
>I'd be interested to see how your parser fares compared to girish's
>work (existing benchmark results, somewhat outdated probably, found at
>http://git.forwardbias.in/?p=qjsonparser.git;a=commit;h=180f3a1ea85adbea4d
>77e50ff3be3e75af918252)
>- those results at least look a lot better than 7 times faster, but
>they are also probably different test data.
I was actually posting results for qjsonparser yesterday (sorry for
wrongly stating it was qjson).
I now redid the benchmarks comparing qjson (as provided by my Ubuntu
install, linked against Qt 4.7), qjsonparser (latest HEAD, linked against
Qt 5) and my qtbinaryjson (linked against Qt5 as well). Here are the
numbers:
qjson:
********* Start testing of tst_Json *********
Config: Using QTest library 4.7.2, Qt 4.7.2
PASS : tst_Json::initTestCase()
RESULT : tst_Json::testByteArray():
16 msecs per iteration (total: 65, iterations: 4)
PASS : tst_Json::testByteArray()
RESULT : tst_Json::testNumbers():
0.19 msecs per iteration (total: 98, iterations: 512)
PASS : tst_Json::testNumbers()
PASS : tst_Json::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped
********* Finished testing of tst_Json *********
qjsonparser:
********* Start testing of tst_Json *********
Config: Using QTest library 5.0.0, Qt 5.0.0
PASS : tst_Json::initTestCase()
RESULT : tst_Json::testByteArray():
1.2 msecs per iteration (total: 83, iterations: 64)
PASS : tst_Json::testByteArray()
RESULT : tst_Json::testString():
1.2 msecs per iteration (total: 80, iterations: 64)
PASS : tst_Json::testString()
RESULT : tst_Json::testNumbers():
0.013 msecs per iteration (total: 57, iterations: 4096)
PASS : tst_Json::testNumbers()
PASS : tst_Json::cleanupTestCase()
Totals: 5 passed, 0 failed, 0 skipped
********* Finished testing of tst_Json *********
qtbinaryjson:
********* Start testing of BenchmarkQtBinaryJson *********
Config: Using QTest library 5.0.0, Qt 5.0.0
PASS : BenchmarkQtBinaryJson::initTestCase()
RESULT : BenchmarkQtBinaryJson::parseNumbers():
0.0054 msecs per iteration (total: 89, iterations: 16384)
PASS : BenchmarkQtBinaryJson::parseNumbers()
RESULT : BenchmarkQtBinaryJson::parseJson():
0.32 msecs per iteration (total: 82, iterations: 256)
PASS : BenchmarkQtBinaryJson::parseJson()
RESULT : BenchmarkQtBinaryJson::parseJsonToVariant():
0.90 msecs per iteration (total: 58, iterations: 64)
PASS : BenchmarkQtBinaryJson::parseJsonToVariant()
PASS : BenchmarkQtBinaryJson::cleanupTestCase()
Totals: 5 passed, 0 failed, 0 skipped
********* Finished testing of BenchmarkQtBinaryJson *********
parseNumbers is parsing the numbers.json file from qjsonparser, the other
tests parse the test.json file from the same source. The file is fully
being loaded into a QByteArray (or QString for one of the qjsonparser
tests), and only the parsing afterwards is being benchmarked.
qjson is clearly slowest. qtbinaryjson is 4 times faster that qjsonparser
and still beats it even when converting the parsed result to a QVariant.
>
>But, parser aside: in general terms, I do think I agree that
>converting to/from QVariant as an "API" isn' all that nice, so your
>approach is a nice alternative.
Thanks.
>
>I do think I'd like to see QDataStream operators on QJsonDocument,
>btw. sending json documents in binary form to/from files or sockets
>would be useful.
That should be very easy to add. Patches welcome :)
Cheers,
Lars
lars....@nokia.com
On 12/11/11 10:39 AM, "ext Sivan Greenberg" <si...@omniqueue.com> wrote:
>Hi Lars!
>
>On Sat, Dec 10, 2011 at 11:32 PM, <lars....@nokia.com> wrote:
>> there were quite a few discussions and requests over the last months to
>> add Json support as part of QtCore, and maybe integrating one of the
>> existing parsers.
>
>There were also discussions and even a couple of bug report about
>adding "native" (a'la making Json first class citizen) support to QML
>as well (I guess this means adding it to the QDeclerativeContext
>engine?) , in favor of having one implementation that'd be standard
>and fast - https://bugreports.qt.nokia.com/browse/QTBUG-12117 .
That might make sense, but would need some further discussion on how to
best do it. I have some ideas, but need to validate this with the QML and
V8 experts :)
>
>>
>> I was however never really happy with them. Mainly because they all
>>parse
>> into a QVariant(Map), and don't offer any real API to create and modify
>> Json. In addition, I believed that it was possible to create a faster
>> parser, and wanted to have a binary representation for the parsed Json.
>>
>I wonder if the QML implementation in the bug report would suffer from
>the same (or perhaps worse?) performance issues?
If it uses qjson behind the scenes it'll be a lot slower than it has to be
(see my post a few minutes ago).
>
>> So I sat down over the last weeks and implemented hacked on something on
>> my own. You can see the current state in our first playground project
>>that
>> Sergio created for me yesterday. You can clone it from
>>
>>ssh://user...@codereview.qt-project.org:29418/playground/qtbinaryjson.gi
>>t
>>
>
>So, could be bind this to have an QML "native" JSON implementation and
>handling? Perhaps also to tie it in to have a JsonModel just as the
>XmlListModel uses XML?
It's possible. The main question will be how to inject it into QML without
converting the data.
>
>> Things look rather promising, and I consider the code to be mostly
>>feature
>> complete. There's work to be done on the API, and documentation is still
>> completely missing, but most of the other pieces are there.
>>
>/me keen to find a spare weekend to check this out.
>
>> What I have right now:
>>
>> * A binary representation of Json, that can be simply mmap'ed from a
>>file,
>> or copied around. This gives us an extremely cheap way to load and save
>> Json objects in binary form.
>
>Would this enable us to store complete binary chunks (files, network
>dumps etc..) as BSON allows? I have some use cases for this if this is
>possible.
You can simply dump the whole Json object to a file and reread it from
there at basically zero cost.
>
>>
>> The binary representation is optimized for fast reading. The binary
>>format
>> is relatively compact and stores the whole object in one blob.
>
>Is this a standard binary format of JSON, or something of your own devise?
It's something I wrote. I wanted it to be something that you can simply
mmap from a file and use without ever having to parse anything to make
saving and restoring of json objects as cheap as possible.
The problem with bson is that it's a streaming format that still requires
parsing and as such was unsuitable for some of the use cases I have.
>
>> There's a bit of work still required to finish it, but I would like to
>> start the discussion about including it into QtCore anyways. I'd also be
>> happy about any feedback.
>
>Many thanks for this!
You're welcome :)
Cheers,
Lars
Robin Burchell
>>Is this a standard binary format of JSON, or something of your own devise?
>
> It's something I wrote. I wanted it to be something that you can simply
> mmap from a file and use without ever having to parse anything to make
> saving and restoring of json objects as cheap as possible.
>
> The problem with bson is that it's a streaming format that still requires
> parsing and as such was unsuitable for some of the use cases I have.
BSON's specification is also a little less generic (& tied to its
origins) than I'd like. Two small examples: having a concept of what a
'symbol' is instead of just sticking to standard js types... and then
there's things like 'timestamp' - which the specification notes is a
"special internal type used by mongodb replication and sharding".
Things like that are, in my opinion, not that great.
Andre Somers
> The binary representation is optimized for fast reading. The binary
> format
> is relatively compact and stores the whole object in one blob.
>> Is this a standard binary format of JSON, or something of your own devise?
> It's something I wrote. I wanted it to be something that you can simply
> mmap from a file and use without ever having to parse anything to make
> saving and restoring of json objects as cheap as possible.
>
> The problem with bson is that it's a streaming format that still requires
> parsing and as such was unsuitable for some of the use cases I have.
>
it is not JSON, then it is "just" some random binary format, right? How
does that help with parsing and providing JSON when you are
communicating with a webservice, for instance? I mean, it is cool to
have a file format that is able to contain a lot of standard stuff and
be quick about it, but I don't really see the connection with JSON, and
calling it a binary format of JSON is confusing to me.
However, like I said, I guess I am missing something here. Could you
explain what the relationship is between this binary format and the JSON
format, and how it helps parsing & creating JSON?
André
Robin Burchell
> However, like I said, I guess I am missing something here. Could you
> explain what the relationship is between this binary format and the JSON
> format, and how it helps parsing & creating JSON?
You're missing the bit where the binary format is not the only
representation. JSON can be parsed into it, it's stored like this in
memory, and it can be serialised back to it (obviously), but it can
also be serialised back to JSON. Or QVariant.
Thiago Macieira
> I now redid the benchmarks comparing qjson (as provided by my Ubuntu
> install, linked against Qt 4.7), qjsonparser (latest HEAD, linked against
> Qt 5) and my qtbinaryjson (linked against Qt5 as well). Here are the
> numbers:
Can you run with -tickcounter -iterations 1024 ?
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center
Intel Sweden AB - Registration Number: 556189-6027
Knarrarnäsgatan 15, 164 40 Kista, Stockholm, Sweden
lars....@nokia.com
wrote:
>On Sunday, 11 de December de 2011 14.47.21, lars....@nokia.com wrote:
>> I now redid the benchmarks comparing qjson (as provided by my Ubuntu
>> install, linked against Qt 4.7), qjsonparser (latest HEAD, linked
>>against
>> Qt 5) and my qtbinaryjson (linked against Qt5 as well). Here are the
>> numbers:
>
>Can you run with -tickcounter -iterations 1024 ?
Sure. Here you go:
qjson:
********* Start testing of tst_Json *********
Config: Using QTest library 4.7.2, Qt 4.7.2
PASS : tst_Json::initTestCase()
RESULT : tst_Json::testByteArray():
40,715,021.989 CPU ticks per iteration (total: 41,692,182,517,
iterations: 1024)
PASS : tst_Json::testByteArray()
RESULT : tst_Json::testNumbers():
559,103.889 CPU ticks per iteration (total: 572,522,383, iterations:
1024)
PASS : tst_Json::testNumbers()
PASS : tst_Json::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped
********* Finished testing of tst_Json *********
qjsonparser:
********* Start testing of tst_Json *********
Config: Using QTest library 5.0.0, Qt 5.0.0
PASS : tst_Json::initTestCase()
RESULT : tst_Json::testByteArray():
3,586,346.742 CPU ticks per iteration (total: 3,672,419,064,
iterations: 1024)
PASS : tst_Json::testByteArray()
RESULT : tst_Json::testString():
3,356,375.772 CPU ticks per iteration (total: 3,436,928,791,
iterations: 1024)
PASS : tst_Json::testString()
RESULT : tst_Json::testNumbers():
39,957.531 CPU ticks per iteration (total: 40,916,512, iterations:
1024)
PASS : tst_Json::testNumbers()
PASS : tst_Json::cleanupTestCase()
Totals: 5 passed, 0 failed, 0 skipped
qtbinaryjson:
********* Start testing of BenchmarkQtBinaryJson *********
Config: Using QTest library 5.0.0, Qt 5.0.0
PASS : BenchmarkQtBinaryJson::initTestCase()
RESULT : BenchmarkQtBinaryJson::parseNumbers():
14,881.212 CPU ticks per iteration (total: 15,238,362, iterations:
1024)
PASS : BenchmarkQtBinaryJson::parseNumbers()
RESULT : BenchmarkQtBinaryJson::parseJson():
853,513.916 CPU ticks per iteration (total: 873,998,250, iterations:
1024)
PASS : BenchmarkQtBinaryJson::parseJson()
RESULT : BenchmarkQtBinaryJson::parseJsonToVariant():
2,495,013.374 CPU ticks per iteration (total: 2,554,893,695,
iterations: 1024)
PASS : BenchmarkQtBinaryJson::parseJsonToVariant()
PASS : BenchmarkQtBinaryJson::cleanupTestCase()
Totals: 5 passed, 0 failed, 0 skipped
********* Finished testing of BenchmarkQtBinaryJson *********
Cheers,
Lars
marius.st...@nokia.com
--
.marius
Sivan Greenberg
> You're missing the bit where the binary format is not the only
> representation. JSON can be parsed into it, it's stored like this in
> memory, and it can be serialised back to it (obviously), but it can
> also be serialised back to JSON. Or QVariant.
Right, and after re-reading over the thread I realized the binary
representation of this specific implementation is just there for sake
of slimness, performance and lightweight exchange.
Attaining to my desire to store large bin chunks in JSON, this means
that as the parser conforms with JSON, it'd store the chunks as base64
encoded strings?
I would love to see something that, given the binary representation
that's already mmap'able, stores the chunks data (streaming media
files,audio, pdfs etc.) in place in its original binary form, and load
each chunk (as is, binary form, no conversions indeed, super fast)
on demand as it is accessed off the JSON model.
-Sivan