Decoding .pb file via protoc command

[tausif...@adplay-mobile.com]

I am new to ProtoBuff file formats. I received a zip file which contains .pb file; the size of the file is approximately 70mb.
I want to decode the file via command line interface. 
I have set up protoc environment variable, library from a zip file named this protoc-3.6.0-win32.zip downloaded from google protobuf documentation site.

I use google ADX, so I have talked with their customer support, for decoding the file, an their replay are as follows:

I've received an update from our specialist team stating that you need to convert the file from protocol buffer to text using protoc. The github for this can be found here- https://github.com/protocolbuffers/protobuf .  This is a command line tool, once you have downloaded it and run a line of code in the command line similar to this: protoc --decode PublisherSettingsList --proto_path . publisher-settings.proto < 2019-03-19T10_00.pb > pub_settings

I have read their documentation and have downloaded the git files; but I don't understand, how is it relevant for decoding my file ?

I have also downloaded the publisher-settings.proto file, from the folowing link:

https://developers.google.com/authorized-buyers/rtb/downloads/publisher-settings-proto

Also, I have tried their commands via cmd, but nothing works.

Can anyone help me regarding this issue ? What is the actual command for decoding the file ? How is it actually done ?

[Adam Cozzette]

Protocol buffers provide a binary format for storing data, but to work with them you also need the relevant schema defined in a .proto file--this would be publisher-settings.proto in your case. Protoc is ordinarily used for generating code to work with protocol buffers but it also has some encoding and decoding functionality built in for convenience.

What error are you getting when you run that protoc command?

Here is another command you could also try: protoc --decode_raw < 2019-03-19T10_00.pb

That version will decode the data as much as possible without relying on the schema, so its output will be less useful but it should at least give you a rough idea of what data is stored.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[tausif...@adplay-mobile.com]

Thanks a lot for your response Adam, but after running your command I am getting the following error:

[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message.  If the message turns out to be larger than 67108864 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf ERROR google/protobuf/io/coded_stream.cc:180] A protocol message was rejected because it was too big (more than 67108864 bytes).  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 67108864
Failed to parse input.

 

As, far as I understood, my file size is too large for decoding. It mentions something about increasing file size limit; but I am not clear how to solve it.

ANY SOLUTIONS ???

[Adam Cozzette]

How big is the actual file?

[tausif...@adplay-mobile.com]

86.2 MB to be exact.

[Adam Cozzette]

That's the surprising that the file is so large and also that its size doesn't match the size in your error message. Can you just confirm that it is a binary file and not a text file? I just want to rule out the possibility that it's formatted in the protobuf text format.

On Tue, Mar 26, 2019 at 9:25 PM tausif.elahi via Protocol Buffers <prot...@googlegroups.com> wrote:

86.2 MB to be exact.

[tausif...@adplay-mobile.com]

Thanks Adam for checking the issue; the file is not binary, as because after opening the file on any text editor(like Sublime) all I can see is different website links and random numbers. For convenience I have uploaded the file on Google drive and you can the check it through this link:

https://drive.google.com/file/d/12sBfxPsG1s2rVSewnkr_2ptfq4102b5M/view?usp=sharing

Furthermore, as I use Google ADX, I have also contacted their support team, and their response are as follows:

 

I've heard back from our Specialist team stating that the error message is related to the capacity of the machine on which you're decoding the file. These files are quite large and take up more space on the system when they're decompressed. Therefore, in order to decode the file, our team has made a few suggestions mentioned below: 

• It appears that you can raise the limit/capacity through "CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h". Searching this lead me to this github. This should guide the technical team in increasing the limit on the machine.
• You can also try decoding the file on the Cloud, which has more capacity than the local machine (and it's easier to add capacity to the cloud versus the local machine)

The github link contains a C++ code, how is the code relevant for decoding my file ?

Things are a bit confusing

[Adam Cozzette]

From looking at that file I see it is in fact protobuf binary format, and it just happens to contain a lot of text strings. Could you try downloading protoc 3.7.1 from here and see if that works?

--

[tausif...@adplay-mobile.com]

Thanks Adam IT WORKED !

Like you said, I used protoc 3.7.1, and I got an output on the command line interface.
I have attached a text file which shows, my output results.

Yet, on the other hand, I have a file size of approx. 186MB, but after running the command I get this output:

Failed to parse input.

 

By the way, is there any command where I can get the output in a text/json file ?

[Adam Cozzette]

Great, I'm glad that worked. Note that you will get more readable results if you use your original --decode command instead of --decode_raw, since --decode_raw interprets the data without having access to the schema.

If this other 186 MB file is failing to parse then I would guess it's not actually a binary protocol buffer.

I don't know if a general utility program exists for converting binary to JSON, but it's fairly easy to write a simple program to do that.

--

[tausif...@adplay-mobile.com]

Thanks Adam, but as I said earlier, I am totally new to this protocol buffer files types and architecture. 
I mentioned about a command provided by google, in the very beginning of this thread, is the command something like that ? Can you kindly provide me the original --decode command ? 
Alongside, is there any sample code that you can provide, for getting an output in any readable format like JSON.
Thanks a lot for your time and efforts

 

[tausif...@adplay-mobile.com]

I appreciate all your effort Adam, but any clue regarding the original decode command, or any sample code for decoding ?

or maybe refer me any documentation ?