C++ json decoding is too slow in 3.1.0

[Sanchay H]

We have a fairly big protobuf (~37MB). Here is the breakup of times:

Serializing proto to binary std::string  ~= 110ms

Serializing proto to json std::string     ~= 4200ms (size ~ 115MB)

Parsing from binary std::string           ~= 700ms

Parsing from json std::string              ~= 35 mins !!!!!

In comparison,

Parsing same json using rapidjson   ~= 800ms

Code I used to serialize and parse:

std::string json;

google::protobuf::util::JsonStringToMessage(json, &proto);

std::string json;

google::protobuf::util::MessageToJsonString(proto, &json);

Surely this is a not as expected right?

[Feng Xiao]

Can you file a github issue? If it takes that long, I suspect the actual problem is that the parser uses up all your machine's physical memory, and causes the system to swap in/out memory pages from/to disk the whole time. 

 

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Sanchay H]

https://github.com/google/protobuf/issues/2305

I don't think it is a swap issue. The process takes 1.45GB and remains there. My system has memory left. Plus there is constant 100% CPU usage.

On Saturday, October 29, 2016 at 5:41:53 PM UTC-7, Feng Xiao wrote:

On Sat, Oct 29, 2016 at 12:57 PM, Sanchay H <sanc...@gmail.com> wrote:

We have a fairly big protobuf (~37MB). Here is the breakup of times:

Serializing proto to binary std::string  ~= 110ms

Serializing proto to json std::string     ~= 4200ms (size ~ 115MB)

Parsing from binary std::string           ~= 700ms

Parsing from json std::string              ~= 35 mins !!!!!

In comparison,

Parsing same json using rapidjson   ~= 800ms

Code I used to serialize and parse:

std::string json;

google::protobuf::util::JsonStringToMessage(json, &proto);

std::string json;

google::protobuf::util::MessageToJsonString(proto, &json);

Surely this is a not as expected right?

Can you file a github issue? If it takes that long, I suspect the actual problem is that the parser uses up all your machine's physical memory, and causes the system to swap in/out memory pages from/to disk the whole time. 

 

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.