Deserializing the "get-entries" response

[Ray Cheng]

Hello,

I started exploring the "get-entries" message and am running into some troubles with de-serializing the response.

I sent this GET request to a pilot log:

https://ct.googleapis.com/aviator/ct/v1/get-entries?start=0&end=1

and I got the response in the expected JSON format:

{"entries":[{"leaf_input":"<base64_encoded_data_1>","extra_data":"<base64_encoded_data_2>"},{"leaf_input":"<base64_encoded_data_3>","extra_data":"<base64_encoded_data_4>"}]}

I then did the following:

1. Base64 decoded the <base64_encoded_data_1> blob into binary data
2. Parse the binary data from step #1 as a MerkleTreeLeaf structure by calling Ct.MerkleTreeLeaf.parseFrom(byte[]) using the generated Java code
   

However, step 2 fails with the exception:

com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).

I then tried the following on the binary data:

protoc --decode_raw 

but I get the message: "Failed to parse input."

At this point, I am thinking that may be I am missing a step as the Base64 decoded value of "leaf_input" does not appear to be a valid protobuf serialized binary data.

I am using protobuf 2.4.1. and I am able to use the generated Java code to serialize and de-serialize structures that I create.

Question:

Am I missing something in trying to get binary data that is parse-able by protobuf?

Thanks.

FWIW, <base64_encoded_data_1> is:

AAAAAAFBbwW1uQAAAAU1MIIFMTCCBBmgAwIBAgIDDjxCMA0GCSqGSIb3DQEBBQUAMDwxCzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5HZW9UcnVzdCwgSW5jLjEUMBIGA1UEAxMLUmFwaWRTU0wgQ0EwHhcNMTMwOTI2MDQ1NDU1WhcNMTQwOTI4MjE0ODA5WjCBvjEpMCcGA1UEBRMgY1IvcENKQnBpUS9PaGpweThnZmoxd0FiQllSU0FsL0MxEzARBgNVBAsTCkdUMjMxNjg2OTMxMTAvBgNVBAsTKFNlZSB3d3cucmFwaWRzc2wuY29tL3Jlc291cmNlcy9jcHMgKGMpMTMxLzAtBgNVBAsTJkRvbWFpbiBDb250cm9sIFZhbGlkYXRlZCAtIFJhcGlkU1NMKFIpMRgwFgYDVQQDEw93d3cuaWNlLXBhYy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDc6SEUQttsmRaubOUXwjHWkQ/agJ15HcOlTjbTvzM1/pRQgp3Vrl+CKFQWPCPH5OyI8xfJQqDzhmB9F5PvISi5kaaq43XA7eZQXVi2bpKcDdQrv3rhCfsQCq4n49IOgRWSA32D1ycACwi3+QH+FPhbDrDhCC6aecm/1yzO0XVbkbywXpm4IjSb3Bb3ZOmUl3c+X6W7PMrut0hxmzn7H8jNXNwT3k6sxW4Ysp/gTnanx+ESnbUB96SYORzzTmiEe2wfafcRxulWg5m40PuauKrWJMVmzNzrlwosQEFDf23BnMj7UD9C80pUQjMbq9UbktIeIp5jAEKKVIlivLLzt8ZxAgMBAAGjggG3MIIBszAfBgNVHSMEGDAWgBRraT1qGEJK3Y8CZTn9NSSGeJEWMDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCcGA1UdEQQgMB6CD3d3dy5pY2UtcGFjLmNvbYILaWNlLXBhYy5jb20wQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL3JhcGlkc3NsLWNybC5nZW90cnVzdC5jb20vY3Jscy9yYXBpZHNzbC5jcmwwHQYDVR0OBBYEFPuWgF6Xk2yEZIMSE9cDO5ATGXvMMAwGA1UdEwEB/wQCMAAweAYIKwYBBQUHAQEEbDBqMC0GCCsGAQUFBzABhiFodHRwOi8vcmFwaWRzc2wtb2NzcC5nZW90cnVzdC5jb20wOQYIKwYBBQUHMAKGLWh0dHA6Ly9yYXBpZHNzbC1haWEuZ2VvdHJ1c3QuY29tL3JhcGlkc3NsLmNydDBMBgNVHSAERTBDMEEGCmCGSAGG+EUBBzYwMzAxBggrBgEFBQcCARYlaHR0cDovL3d3dy5nZW90cnVzdC5jb20vcmVzb3VyY2VzL2NwczANBgkqhkiG9w0BAQUFAAOCAQEAJJGA2PDjRjjhsbcy07DwR5JzIZoGQH5WxaZOQzvHRG+XhQFAuKiWZKBH4kB2exF3HNWi8N5dU+xxiryVUYoluLocu3zaGpUAAdtfUWg9lMZi5dSBJRZCQSywAofUJ0t8uvLwZe9eryCDGQHchHGId3ILwdt064RYgAgqxwXWK58mUpNQ5HB4Ln/N/AF8IRcSM3lA7R2vVPtuXCbHMfYP0re+qdAT/tsgFp4own10Fpl94O4Epr+p95RF1orAPSE4MLjYHjRUMi58fV+dafNQbfbFZ4zLAvhrVZHhle2tsuQK08X0rFYSi0Wt5PzmD17/Z3l4ChwofBLJtZ/8lDe+YAAA

[Tom Fitzhenry]

Hi Ray,

MerkleTreeLeaf is not encoded using protobuf. It is encoded as a packed struct, so a suitable class might be java.io.DataInputStream .

It took a while for me to realise this. I found https://github.com/agl/certificatetransparency/blob/a306835b12ca1a49506e96b6706431dc7f296af9/db.go#L255 useful when doing this, for what it's worth.

Regards,
Tom

[Eran Messeri]

The exact MerkleTreeLeaf is described in section 3.4 of the rfc. The data structures are defined according to section 4 of RFC5246.

An example of how to unpack these structures can be seen here:

https://code.google.com/p/certificate-transparency/source/browse/src/python/ct/client/tls_message.py

In this code the tls-encoded struct is read into a ProtoBuf defined in client.proto.

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ray.ch...@gmail.com]

Tom and Eran,

Thanks very much for your clarification on the response encoding and the pointers to the sample code.

I am now able to de-serialize the various get-entries response components.

Regards

Ray

On Tuesday, February 25, 2014 5:33:11 AM UTC-5, Eran Messeri wrote:
> The exact MerkleTreeLeaf is described in section 3.4 of the rfc. The data structures are defined according to section 4 of RFC5246.
>
>
>
> An example of how to unpack these structures can be seen here:
> https://code.google.com/p/certificate-transparency/source/browse/src/python/ct/client/tls_message.py
>
>
>
>
> In this code the tls-encoded struct is read into a ProtoBuf defined in client.proto.
>
>
>
>
> On Tue, Feb 25, 2014 at 9:02 AM, Tom Fitzhenry <goo...@tom-fitzhenry.me.uk> wrote:
>
>
> Hi Ray,
>
> MerkleTreeLeaf is not encoded using protobuf. It is encoded as a packed struct, so a suitable class might be java.io.DataInputStream .
>
>
> It took a while for me to realise this. I found https://github.com/agl/certificatetransparency/blob/a306835b12ca1a49506e96b6706431dc7f296af9/db.go#L255 useful when doing this, for what it's worth.
>
>
> Regards,
> Tom
>
>
>
> On Monday, 24 February 2014 20:55:40 UTC, Ray Cheng wrote:
>
> Hello,
>
>
> I started exploring the "get-entries" message and am running into some troubles with de-serializing the response.
>
>
>
> I sent this GET request to a pilot log:
>
> https://ct.googleapis.com/aviator/ct/v1/get-entries?start=0&end=1
>
>
>
> and I got the response in the expected JSON format:
>
> {"entries":[{"leaf_input":"<base64_encoded_data_1>","extra_data":"<base64_encoded_data_2>"},{"leaf_input":"<base64_encoded_data_3>","extra_data":"<base64_encoded_data_4>"}]}
>
>
>
>
> I then did the following:

> Base64 decoded the <base64_encoded_data_1> blob into binary dataParse the binary data from step #1 as a MerkleTreeLeaf structure by calling Ct.MerkleTreeLeaf.parseFrom(byte[]) using the generated Java code