[boost] Json read...segment by segment

[vijay...@edisphere.com]

Hi,



I am using below code to read load Json file in boost::property_tree::ptree
object. Function read_json(.) read all file content at a time. I want to
know is there any way to read file content segment by segment.



Thanks and regards,

Vijay





int _tmai3n(int argc, _TCHAR* argv[])

{

try

{

std::stringstream ss;



std::ifstream
file("D:\\EdisphereComp\\kit\\1003_Kit\\1003\\Inputfiles\\AI98_Test_3.json",
std::ios::binary);



// Make sure we have something to read.

if (!file.is_open()) {

throw (std::exception("Could not open file."));

}



// Copy contents "as efficiently as possible".

ss << file.rdbuf();





boost::property_tree::ptree pt;

boost::property_tree::read_json(ss, pt);






boost::property_tree::write_json("D:\\EdisphereComp\\kit\\1003_Kit\\1003\\In
putfiles\\AI98_Test_3_sprite.json", pt);





}

catch (boost::property_tree::json_parser_error e)

{

;

}



//}

return 0;

}




_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Sebastian Redl]

On 06.07.2015 08:45, vijay...@edisphere.com wrote:
> Hi,
>
>
>
> I am using below code to read load Json file in boost::property_tree::ptree
> object. Function read_json(.) read all file content at a time. I want to
> know is there any way to read file content segment by segment.
>

The PropertyTree parser is not a streaming parser, it builds a tree out
of the entire contents of a file. This cannot be changed. I also don't
know of any DOM parser for any semi-structured language that supports
this; I wouldn't know how to do this in a useful way.

Can you explain your use case in more detail perhaps?

Sebastian

[Bjorn Reese]

On 07/06/2015 09:55 AM, Sebastian Redl wrote:

> The PropertyTree parser is not a streaming parser, it builds a tree out
> of the entire contents of a file. This cannot be changed. I also don't
> know of any DOM parser for any semi-structured language that supports
> this; I wouldn't know how to do this in a useful way.

I am working on a streaming JSON parser [1] (documentation is still
missing though.) One of the examples [2] is a reimplementation of
property_tree::read_json(). It currently parses the entire content,
but only because it mimics the original behavior. It could be rewritten
to build the property_tree incrementally.

[1] https://github.com/breese/protocol
[2]
https://github.com/breese/protocol/tree/master/example/json/property_tree

[Sebastian Redl]

On 06.07.2015 10:55, Bjorn Reese wrote:
> On 07/06/2015 09:55 AM, Sebastian Redl wrote:
>
>> The PropertyTree parser is not a streaming parser, it builds a tree out
>> of the entire contents of a file. This cannot be changed. I also don't
>> know of any DOM parser for any semi-structured language that supports
>> this; I wouldn't know how to do this in a useful way.
>
> I am working on a streaming JSON parser [1] (documentation is still
> missing though.) One of the examples [2] is a reimplementation of
> property_tree::read_json(). It currently parses the entire content,
> but only because it mimics the original behavior. It could be rewritten
> to build the property_tree incrementally.

I just replaced the internal parser of PropertyTree (the old one was a
Spirit.Classic parser) with a hand-written streaming parser (push, while
yours appears to be pull) too. It's on the develop branch. The real
problem is a sensible way to mark a tree as partial. Who wants to work
with such a structure? In such cases, I'm pretty sure it would be
simpler to use the streaming interface directly.

Sebastian

[Boris Rasin]

On 7/6/2015 12:06 PM, Sebastian Redl wrote:
> I just replaced the internal parser of PropertyTree (the old one was a
> Spirit.Classic parser) with a hand-written streaming parser (push,
> while yours appears to be pull) too. It's on the develop branch.

Will this remove the requirement of boost/thread for PropertyTree JSON
reader/writer in multi-threaded environments? When do you plan to
release it?

[Sebastian Redl]

On 06.07.2015 16:47, Boris Rasin wrote:
> On 7/6/2015 12:06 PM, Sebastian Redl wrote:
>> I just replaced the internal parser of PropertyTree (the old one was
>> a Spirit.Classic parser) with a hand-written streaming parser (push,
>> while yours appears to be pull) too. It's on the develop branch.
>
> Will this remove the requirement of boost/thread for PropertyTree JSON
> reader/writer in multi-threaded environments?

It will fix the multithreading bug #5520, if that's what you're asking.
PropertyTree doesn't have any direct dependency on Thread.

https://svn.boost.org/trac/boost/ticket/5520

> When do you plan to release it?

The test matrix is going green, so it should be in 1.59. But I hope that
the new Boost.Test will be merged to master first, since a workaround in
my test cases is affected by an internals change between the old and new
version.

Sebastian

[Boris Rasin]

On 7/6/2015 6:20 PM, Sebastian Redl wrote:
> It will fix the multithreading bug #5520, if that's what you're
> asking. PropertyTree doesn't have any direct dependency on Thread.
>
> https://svn.boost.org/trac/boost/ticket/5520

Yes, that's what I was asking. Currently, to use PropertyTree JSON
reader/writer in multi-threaded environment (even if each ptree object
is only accessed by single thread) one has to define
BOOST_SPIRIT_THREADSAFE which brings boost/thread in. That pretty much
makes use of PropertyTree JSON reader/writer in multi-threaded
environment dependent on boost/thread.

Anyway, I'm looking forward to new version, and thanks for the great work.

[Bjorn Reese]

On 07/06/2015 11:06 AM, Sebastian Redl wrote:

> I just replaced the internal parser of PropertyTree (the old one was a
> Spirit.Classic parser) with a hand-written streaming parser (push, while
> yours appears to be pull) too. It's on the develop branch. The real

Yes, mine is a pull parser. Pull parsers are more versatile than push
parsers, and therefore better suited as building-blocks.

For instance, it is trivial to create a push parser with a pull parser.
It is just a loop and switch as can be seen in this example:


https://github.com/breese/protocol/blob/master/example/json/push_parser/push_parser.hpp

Try creating a push parser with a pull parser ;-)

It is also fairly easy to create Boost.Serialization archives with pull
parsers.

> problem is a sensible way to mark a tree as partial. Who wants to work
> with such a structure? In such cases, I'm pretty sure it would be
> simpler to use the streaming interface directly.

Agreed.