Discovering unlogged certificates in internet-wide scans

[Tim Smith]

Hi MDSP,

I went looking for corpuses of certificates that may not have been
previously logged to CT and found some in the Rapid7 "More SSL" dataset,
which captures certificates from their scans of non-HTTPS ports for
TLS-speaking services.

I wrote up some findings at
http://blog.tim-smith.us/2018/03/moressl-spelunking/.

A few highlights include:
- of the ~10 million certificates in the corpus, about 20% had valid
signatures and chained to roots included in the Mozilla trust store
- about 50,000 of the 2 million trusted certificates had not previously
been logged
- about half of the novel certificates were unexpired

There were interesting examples of unexpired, non-compliant, trusted
certificates chaining to issuers including GoDaddy, NetLock, Logius, and
Entrust. (I have not taken any action to inform issuers of these findings,
other than this message and by publishing the certificates to CT logs.)

I welcome any feedback or questions about the value of the approach and the
findings.

Thanks,
Tim

[Kurt Roeckx]

On Sat, Mar 31, 2018 at 10:14:27PM +0000, Tim Smith via dev-security-policy wrote:
> Hi MDSP,
>
> I went looking for corpuses of certificates that may not have been
> previously logged to CT and found some in the Rapid7 "More SSL" dataset,
> which captures certificates from their scans of non-HTTPS ports for
> TLS-speaking services.

Have you done the for their other scans?

There are more people that do such regular scans, and I wish they
all logged those certs as soon as they saw them.


Kurt

[Tim Smith]

On Sat, Mar 31, 2018 at 3:26 PM, Kurt Roeckx <ku...@roeckx.be> wrote:
> Have you done the for their other scans?

I haven't. The Rapid7 HTTPS corpus is much larger; I'm not sure my
approach will scale that far and I imagine the new discovery rate will
be lower.

Censys has been interested in submitting new certificates to CT in the
past [1]; I wonder if they've resumed.

Tim

[1] https://groups.google.com/a/censys.io/d/msg/discussion/nrbN70xegEs/dmbunh7jAgAJ

[Alex Cohn]

I'm currently grabbing certs from Censys's BigQuery extracts and
submitting them to the Argon logs (and Daedalus/Rocketeer for certs
that fall before/after Argon's not-after range).

There's a fair bit of latency in the process; I'm only running this
script weekly (it costs about $4 a pop in BigQuery usage alone) and
Censys only updates BigQuery every couple days. But there are only a
handful of certs in the Censys corpus as of a couple weeks ago that
are not also in CT. [1]

I've talked with Censys about them doing this directly, and my
impression was that it's something they're in support of but not
something they have the bandwidth to do themselves right now.

Alex

[1] https://censys.io/certificates?q=metadata.added_at%3A%5B*+TO+2018-03-15%5D+and+not+tags.raw%3Act+and+validation.google_ct_primary.valid%3A+true

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy

[Michael Casadevall]

On 03/31/2018 06:14 PM, Tim Smith via dev-security-policy wrote:
> Hi MDSP,
>
> I went looking for corpuses of certificates that may not have been
> previously logged to CT and found some in the Rapid7 "More SSL" dataset,
> which captures certificates from their scans of non-HTTPS ports for
> TLS-speaking services.
>

Pretty interesting read, and always happy to see more information go
into CT. One thing I couldn't divine from your data was how did you look
for non-HTTPS services? Did you port scan and do service discovery, or
did you simply knock on well known ports that either are SSL by default
or support a STARTTLS equivalent?

If so, which well known ports were knocked on?

> I wrote up some findings at
> http://blog.tim-smith.us/2018/03/moressl-spelunking/.
>
> A few highlights include:
> - of the ~10 million certificates in the corpus, about 20% had valid
> signatures and chained to roots included in the Mozilla trust store
> - about 50,000 of the 2 million trusted certificates had not previously
> been logged
> - about half of the novel certificates were unexpired
>

When you say roots included to Mozilla trust store, how was this used
exactly? I see you used X509Validator, but did you just throw all the
NSS PEMs or did you remove the ones that are technically constrained?
(i.e., CNNIC is distrusted for new certificates, but is in the root
store for existing certificates before technical restrictions were applied)

This could influence the number of valid or invalid certificates you
saw. I'd also be interested in see or graph that shows how many
certificates chain up to a given root.

I do wonder if there would be value in a tool that basically takes a
x509 certificate in, and then runs it through NSS to determine validity
applying all technical constraints upon the way. I might have to fiddle
with NSS to see how hard it is to get at that functionality.

> There were interesting examples of unexpired, non-compliant, trusted
> certificates chaining to issuers including GoDaddy, NetLock, Logius, and
> Entrust. (I have not taken any action to inform issuers of these findings,
> other than this message and by publishing the certificates to CT logs.)
>

Given all these certificates are new to CT, I'm guessing none of them
have come up before on MDSP.

Thanks for your effort,
Michael

[Tim Smith]

On Sat, Mar 31, 2018 at 6:28 PM, Michael Casadevall via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
> Pretty interesting read, and always happy to see more information go
> into CT. One thing I couldn't divine from your data was how did you look
> for non-HTTPS services? Did you port scan and do service discovery, or
> did you simply knock on well known ports that either are SSL by default
> or support a STARTTLS equivalent?
>
> If so, which well known ports were knocked on?

Thanks for taking a look. My understanding of Rapid7's methodology [1,
2] is that they knock on well-known ports. The services they emit data
for are pop3/s (110, 995), nntps (563), mqtt (8833), ftps (990),
smtp/s (25, 465, 587), and imap/s (143, 990).

I consumed their published data set and didn't perform any scanning myself.

> When you say roots included to Mozilla trust store, how was this used
> exactly? I see you used X509Validator, but did you just throw all the
> NSS PEMs or did you remove the ones that are technically constrained?

I used the certifi distribution [3], which is aware of "included but
distrusted" [4]. The certificates were "validated" naively without
considering e.g. path length or name constraints; certificates failing
those constraints would certainly be interesting but hopefully rare.

> I'd also be interested in see or graph that shows how many
> certificates chain up to a given root.

Sure; would you like to see that because it would help sanity-check
the data, or because it might reveal differences vs. certificates used
for HTTPS, or some other reason?

Thanks,
Tim

[1] https://github.com/rapid7/sonar/wiki/More-SSL-Certificates
[2] https://scans.io/study/sonar.moressl
[3] https://certifiio.readthedocs.io/en/latest/
[4] https://github.com/certifi/extract-nss-root-certs

[Michael Casadevall]

On 03/31/2018 09:53 PM, Tim Smith wrote:
> On Sat, Mar 31, 2018 at 6:28 PM, Michael Casadevall via
> dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
> Thanks for taking a look. My understanding of Rapid7's methodology [1,
> 2] is that they knock on well-known ports. The services they emit data
> for are pop3/s (110, 995), nntps (563), mqtt (8833), ftps (990),
> smtp/s (25, 465, 587), and imap/s (143, 990).
>
> I consumed their published data set and didn't perform any scanning
myself.
>

Ah, I misread, I thought you had done the scanning directly. Still, it
was good to take a look at this, and I'm looking through Rapid7's docs
on how to play with the data set myself.

>> When you say roots included to Mozilla trust store, how was this used
>> exactly? I see you used X509Validator, but did you just throw all the
>> NSS PEMs or did you remove the ones that are technically constrained?
>
> I used the certifi distribution [3], which is aware of "included but
> distrusted" [4]. The certificates were "validated" naively without
> considering e.g. path length or name constraints; certificates failing
> those constraints would certainly be interesting but hopefully rare.
>

Hrm, I might have a weekend project here. I've always been curious to
what extent technical constraints actually affected things. I may need
to download their data set and play with it somewhat.

>> I'd also be interested in see or graph that shows how many
>> certificates chain up to a given root.
>
> Sure; would you like to see that because it would help sanity-check
> the data, or because it might reveal differences vs. certificates used
> for HTTPS, or some other reason?
>

Partially for non-HTTPS usage, but I also wanted to see the
cross-section of how much "diversity" there was in how much of the
internet chains of a specific root certificate.

Michael

[Eric Mill]

Did you submit the ~25K unexpired unlogged certs to CT?

On Sat, Mar 31, 2018 at 6:14 PM, Tim Smith via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Hi MDSP,
>
> I went looking for corpuses of certificates that may not have been
> previously logged to CT and found some in the Rapid7 "More SSL" dataset,
> which captures certificates from their scans of non-HTTPS ports for
> TLS-speaking services.
>

> I wrote up some findings at
> http://blog.tim-smith.us/2018/03/moressl-spelunking/.
>
> A few highlights include:
> - of the ~10 million certificates in the corpus, about 20% had valid
> signatures and chained to roots included in the Mozilla trust store
> - about 50,000 of the 2 million trusted certificates had not previously
> been logged
> - about half of the novel certificates were unexpired
>

> There were interesting examples of unexpired, non-compliant, trusted
> certificates chaining to issuers including GoDaddy, NetLock, Logius, and
> Entrust. (I have not taken any action to inform issuers of these findings,
> other than this message and by publishing the certificates to CT logs.)
>

> I welcome any feedback or questions about the value of the approach and the
> findings.
>
> Thanks,
> Tim

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

--
konklone.com | @konklone <https://twitter.com/konklone>

[Daymion Reynolds]

As an FYI only:

We did review the one cert cited below for term length. The certificate was issued in 2013 before the current max term duration was defined. This cert is grandfathered in and does not require revocation. In May of this year it expires.

regards,
Daymion

[Tim Smith]

On Mon, Apr 9, 2018 at 9:46 AM Daymion Reynolds via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> As an FYI only:
>
> We did review the one cert cited below for term length. The certificate
> was issued in 2013 before the current max term duration was defined. This
> cert is grandfathered in and does not require revocation. In May of this
> year it expires.
>

Hello,

The certificate [1] has a notBefore date of Apr 19, 2013. Version 1.1.3 of
the BRs [2], effective from Feb 21, 2013, required that new certificates
must not have a validity period greater than 60 months (section 9.4, p 12).
The delta between notAfter and notBefore is greater than 60 months, so it
was not compliant at the time of issuance, unless the notBefore date does
not reflect the date of issuance.

Thanks,
Tim

[1] https://crt.sh/?id=370273130&opt=cablint,ocsp
[2] https://cabforum.org/wp-content/uploads/Baseline_Requirements_V1_1_3.pdf

[Stephen Davidson]

Hello,



Many thanks for the research - this CT analysis is both fascinating and useful. I'd like to address the following statement:



"Noncompliance already visible from previously logged certificates. The HydrantID SSL ICA G2 CA is trusted by Mozilla (via QuoVadis) for TLS authentication, but issues certs intended for IPSEC and which lack serverAuth and clientAuth EKU values, which are not BR-compliant (7.1.2.3.f)."



For the sake of reference, here's a link to one of the certificates referenced: https://crt.sh/?id=380352272&opt=zlint. Responding to the two points:



1. IPSEC EKU

These certificates contain extKeyUsage values for IPSEC due to a customer requirement. This is allowed under the Baseline Requirements:



"7.1.2.3.f. extKeyUsage (required) Either the value id-kp-serverAuth [RFC5280] or id-kp-clientAuth [RFC5280] or both values MUST be present. id-kp-emailProtection [RFC5280] MAY be present. Other values SHOULD NOT be present."



RFC 2119 defines SHOULD NOT as:



4. SHOULD NOT
This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.



2. serverAuth and clientAuth EKU

These certificates are compliant with the BR and contain the required extKeyUsage values for both id-kp-serverAuth or id-kp-clientAuth. It appears that zLint "w_sub_cert_eku_extra_values" provides an incomplete message relating to the BR 7.1.2.3:



*****************************************

* ZLint showing just w_sub_cert_eku_extra_values description

*****************************************

$ zlint -list-lints-json | grep 'w_sub_cert_eku_extra_values'

{"name":"w_sub_cert_eku_extra_values","description":"Subscriber Certificate: extKeyUsage either the value id-kp-serverAuth or id-kp-clientAuth or both values MUST be present.","citation":"BRs: 7.1.2.3"}



This lint shows a warning due to the inclusion of additional extKeyUsage values. The zLint description ideally should include the "Other values SHOULD NOT be present" portion of the BR to reflect that the warning may be generated even though the cert is compliant with the BR.



Regards,

Stephen Davidson

QuoVadis

-----Original Message-----
From: dev-security-policy <dev-security-policy-bounces+s.davidson=quovadisg...@lists.mozilla.org> On Behalf Of Tim Smith via dev-security-policy
Sent: Saturday, March 31, 2018 7:15 PM
To: Mozilla <mozilla-dev-s...@lists.mozilla.org>
Subject: Discovering unlogged certificates in internet-wide scans



Hi MDSP,



I went looking for corpuses of certificates that may not have been previously logged to CT and found some in the Rapid7 "More SSL" dataset, which captures certificates from their scans of non-HTTPS ports for TLS-speaking services.



I wrote up some findings at

http://blog.tim-smith.us/2018/03/moressl-spelunking/.



A few highlights include:

- of the ~10 million certificates in the corpus, about 20% had valid signatures and chained to roots included in the Mozilla trust store

- about 50,000 of the 2 million trusted certificates had not previously been logged

- about half of the novel certificates were unexpired



There were interesting examples of unexpired, non-compliant, trusted certificates chaining to issuers including GoDaddy, NetLock, Logius, and Entrust. (I have not taken any action to inform issuers of these findings, other than this message and by publishing the certificates to CT logs.)



I welcome any feedback or questions about the value of the approach and the findings.



Thanks,

Tim

_______________________________________________

dev-security-policy mailing list

dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>

https://lists.mozilla.org/listinfo/dev-security-policy

[Tim Smith]

Hi Stephen,

Thank you for the correction; I regret the error.

On Tue, Apr 10, 2018 at 8:12 AM Stephen Davidson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> These certificates are compliant with the BR and contain the required
> extKeyUsage values for both id-kp-serverAuth or id-kp-clientAuth. It
> appears that zLint "w_sub_cert_eku_extra_values" provides an incomplete
> message relating to the BR 7.1.2.3:
>

I opened a pull request against zlint to fix the error message for the
extra_values lint: https://github.com/zmap/zlint/pull/218

I'll update the blog post momentarily.

Tim