WoSign log failure to incorporate entry within the MMD

233 views
Skip to first unread message

Graham Edgecombe

unread,
Dec 15, 2017, 1:28:56 PM12/15/17
to ct-p...@chromium.org
Hi,

My monitor's SCT feedback endpoint has received a certificate/SCT pair
(no idea who from - whoever it was deserves the credit!) with a valid
signature from WoSign's https://ctlog.wosign.com log that, at the time
of writing, has not been included in the log. The SCT has a timestamp of
2016-09-21 17:46:58, so it's well over a year old.

The SCT is for a precert entry, which makes verifying the signature a
bit of a pain, but it is indeed valid. The leaf_input_hash is:

FjB8ur40yIP8ixl2JegRSsdxodT8UUPs1mW5E9g4I4Y=

I've confirmed the signature validation and leaf_input_hash with two
separate implementations (my own, and a quick script I hacked on top of
the reference Go library), so I'm fairly sure that I haven't made a
mistake. The script also works on Aviator/Rocketeer SCTs for the same
certificate, which do have valid inclusion proofs :)

At the time of writing, fetching
https://ctlog.wosign.com/ct/v1/get-proof-by-hash?hash=FjB8ur40yIP8ixl2JegRSsdxodT8UUPs1mW5E9g4I4Y%3D&tree_size=7770737
returns a "couldn't find hash" error. My monitor also keeps a copy of
all entries locally, and none of them have a matching leaf_input_hash -
so I don't think it's merely a problem with the get-proof-by-hash
endpoint.

I've attached the certificate chain and the SCT (in binary format) to
this message. I've also attached the current STH in case the entry is
subsequently added to the log. This should be sufficient to reproduce
the steps I took above.

Graham
chain.pem
sct.b64
sth.json

Kat Joyce

unread,
Dec 18, 2017, 3:20:14 AM12/18/17
to Graham Edgecombe, Certificate Transparency Policy
Hi Graham and ct-policy,

I have verified Graham's findings.  

I believe that the precertificate in question is this one: https://crt.sh/?id=33571166, and the SCT that he refers to is the Wosign one present in the corresponding leaf certificate: https://crt.sh/?id=250873976.

Kat


Graham

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20171214233429.lulgwfjaangrrd3e%40rictusempra.grahamedgecombe.com.

Andrew Ayer

unread,
Dec 18, 2017, 11:38:11 AM12/18/17
to Graham Edgecombe, ct-p...@chromium.org
I have audited every single embedded SCT found in public Certificate
Transparency logs. I found 18 pre-certificate SCTs that WoSign (log
ID QbLcLonmPOSvG6e7Kb9oxt7m+fHMBH4w3/rjs7olkmM=) failed to incorporate
(including the one reported by Graham).

I've attached evidence of these SCTs. It's a JSON array of objects
containing:

* sct: The SCT (in JSON, per RFC6962 Section 4.1)
* precert:
* issuer_key_hash: the issuer key hash, in base64
* tbs_certificate: the pre-certificate TBSCertificate, in base64

Regards,
Andrew
wosign_bad_scts.json

WoTrus log operator

unread,
Dec 19, 2017, 3:50:46 AM12/19/17
to Certificate Transparency Policy
Hi, I'm WoTrus CT Log operator.

If there is an SCT without exist in merkle tree, there are two possibilities:
1. log private key leak, hacker can use Log's private key to sign sct
2. Program error

After analyse and carefully check our servers,we believe this problem was not cause by Log's private key leak.

So, it was caused by our log program.

WoTrus use the source code of CT Log from: https://github.com/google/certificate-transparency.git
the version of source code is 7e1cf3b2ceddc0d825af214645b818ca84b1cbc2
the version of etcd is 2.0.9, which we get from https://github.com/coreos/etcd/releases/download/v2.0.9/etcd-v2.0.9-linux-amd64.tar.gz

And after analyse the source code, I think that if the entry data has been existed on \entry directory of etcd, certificate entry will incorporate within the Log certainly.
It look like after after data had sent to \entry directory on etcd, which didn't appear on etcd.

But now we don't have direct proof, the preliminary conclusions is come from CT Log source code analyse because we cann't find any special on etcd 's log.

you can find CT Log add cert flow on here:
https://github.com/google/certificate-transparency/blob/master/docs/DesignDoc.md#add-chain--add-pre-chain

WoTrus log operator

unread,
Dec 19, 2017, 4:05:46 AM12/19/17
to Certificate Transparency Policy
If there is any errors on my analyse, thanks for point out.

WoTrus log operator

unread,
Dec 19, 2017, 4:52:35 AM12/19/17
to Certificate Transparency Policy
And now, we had update our etcd to version 2.1.3.

Ryan Sleevi

unread,
Dec 19, 2017, 5:10:12 PM12/19/17
to WoTrus log operator, Certificate Transparency Policy
I'm not sure that this is a particularly inspiring post-mortem. It sounds like you don't have logs to determine what happened, so you're approximating - is that correct?

If you don't have logs at that time, what makes you believe it wasn't a private key leak?

On Tue, Dec 19, 2017 at 4:05 AM, WoTrus log operator <liang...@gmail.com> wrote:
If there is any errors on my analyse, thanks for point out.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

Kane York

unread,
Dec 20, 2017, 4:51:01 AM12/20/17
to Certificate Transparency Policy


On Tuesday, December 19, 2017 at 12:50:46 AM UTC-8, WoTrus log operator wrote:
Hi, I'm WoTrus CT Log operator.
...

And after analyse the source code, I think that if the entry data has been existed on \entry directory of etcd, certificate entry will incorporate within the Log certainly.
It look like after after data had sent to \entry directory on etcd, which didn't appear on etcd.

But now we don't have direct proof, the preliminary conclusions is come from CT Log source code analyse because we cann't find any special on etcd 's log.

To paraphrase, you sent the data and it didn't show up. Or you're pretty sure that's what happened.

And now, we had update our etcd to version 2.1.3.

And the only thing you've done to fix this is to update etcd? That isn't a comprehensive resolution at all! Do you have any solid reason to believe that updating etcd will prevent this problem from recurring?

In the list posted by Andrew, the timestamps are mostly from December 2016 to January 2017. However, Graham's post is a certificate from September 2016. This is odd - most software errors are continuous; where's the failures from November? was the issue fixed in January 2017?

etcd v2.0.9 was released in April 2015, (source: https://github.com/coreos/etcd/releases?after=v2.1.0-rc.0, scroll down) so I find it very doubtful that etcd updates have anything to do with this, seeing 2.0.13 was released in June 2015, far before this January 2017 tentative end of timeframe for these failures (I find myself doubting whether the issue actually stopped this January).

And that's all the labor you're getting out of me. Someone should check whether Andrew actually found all the dropped SCTs, since it seems like you didn't?

Respectfully,
Kane
 
Reply all
Reply to author
Forward
0 new messages