.tg Certificates Issued by Let's Encrypt

[Daniel Cater]

I notice that on https://crt.sh/mozilla-onecrl there are lots of certificates that have recently been added to OneCRL from the .tg TLD (Togo), including ones for high-profile domains such as google.tg. The issuances occurred 3 days ago, on 1st November.

I don't see a thread already for this here, or on https://letsencrypt.org/blog/ so I thought I would start one.

>From the check-in comment "registry problems", I assume that this is a problem with the TLD rather than with Let's Encrypt.

As OneCRL and CRLSets are public this information is being noticed. There is likely a large overlap between the people that read this group and the people that monitor those lists. That said, be mindful of posting any specific technical vulnerabilities or exploits which may not yet be patched.

[Kathleen Wilson]

As you have noticed based on OneCRL and crt.sh, there was a problem with
the *.tg registry, and SSL certificates were issued to domains in *.tg
that probably should not have been issued. As you can see, the Let's
Encrypt CA was made aware of the problem and has already responded by
revoking the impacted certs, and we have added entries for those certs
to OneCRL. Unfortunately, the CT data shows that other CAs also recently
issued certs containing *.tg domains.

I have not personally spoken with the people at the *.tg registry yet,
but my understanding is that the problem has been fixed on their end.

This is a new scenario to me -- having a problem at a registry that
results in SSL certs being issued that otherwise would not have been
issued. So I am trying to figure out how to respond to it. For example,
should I send email to only the CAs who are showing up in CT and crt.sh
as having issued SSL certs for the *.tg TLD within the past few days? Or
should I send an email blast out to all CAs in Mozilla's program?

I think those CAs need to re-validate their recently issued SSL certs
that contain any *.tg domains, and possibly revoke such certs and send
us the info so corresponding entries can be added to OneCRL. But, as
this is new to me, I will appreciate thoughtful and constructive input
in this.

Thanks,
Kathleen

[Daniel Cater]

I think it depends on whether the issue has been fixed or not. If it has not been fixed, then I would say that all CAs need to put a hold on .tg certificate issuance as a priority. If a registry can be compromised, then potentially the integrity of all 10 blessed methods is at risk.

If it has been fixed, then I think revalidating all recent issuances (for some definition of recent) makes sense, and revoking those that cannot be confirmed as legitimate.

CAA might have helped in some situations here, for example google.tg doesn't have a CAA record and if it referenced "pki.goog" then it's unlikely an attacker could have gained a valid certificate from that CA. In a lot of instances I expect it wouldn't have helped, because the attacker could just go to the approved CA and request the certificate, and the domain validation would succeed using whatever registry/domain-takeover method was used (unless the CA has extra checkpoints / blocks in place for particular high-profile domains).

I note that .tg still doesn't have DNSSEC enabled: http://stats.research.icann.org/dns/tld_report/

This may or may not have helped mitigate the attack for those sites, depending on the technical details.

[Daniel Cater]

Hmm, CAA records could also potentially be spoofed in this situation, in which case they would also not be trustworthy (save for cached records with a long TTL).

[Ben Laurie]

On 4 November 2017 at 19:54, Kathleen Wilson via dev-security-policy <

dev-secur...@lists.mozilla.org> wrote:

> On 11/4/17 5:36 AM, Daniel Cater wrote:
>

> As you have noticed based on OneCRL and crt.sh, there was a problem with
> the *.tg registry, and SSL certificates were issued to domains in *.tg that
> probably should not have been issued. As you can see, the Let's Encrypt CA
> was made aware of the problem and has already responded by revoking the
> impacted certs, and we have added entries for those certs to OneCRL.
> Unfortunately, the CT data shows that other CAs also recently issued certs
> containing *.tg domains.
>
> I have not personally spoken with the people at the *.tg registry yet, but
> my understanding is that the problem has been fixed on their end.
>

> This is a new scenario to me -- having a problem at a registry that
> results in SSL certs being issued that otherwise would not have been
> issued. So I am trying to figure out how to respond to it. For example,
> should I send email to only the CAs who are showing up in CT and crt.sh as
> having issued SSL certs for the *.tg TLD within the past few days? Or
> should I send an email blast out to all CAs in Mozilla's program?
>

> I think those CAs need to re-validate their recently issued SSL certs that
> contain any *.tg domains, and possibly revoke such certs and send us the
> info so corresponding entries can be added to OneCRL. But, as this is new
> to me, I will appreciate thoughtful and constructive input in this.

Since CT is not (yet) compulsory, it seems you probably have to contact all
CAs, doesn't it?

[Kathleen Wilson]

On 11/6/17 3:40 AM, Ben Laurie wrote:
> Since CT is not (yet) compulsory, it seems you probably have to contact all
> CAs, doesn't it?
>

To close the loop on this...

I have added the following to the draft of the November 2017 CA
Communication.

~~
ACTION 8: Check for issuance of TLS/SSL certificates to .tg domains from
October 25 to November 2, 2017.

We believe that the .tg Registry was compromised from October 25 to
November 1, 2017, such that a perpetrator set the Name Server (NS)
Records for some domains to name servers controlled by them, and then
successfully obtained SSL certificates for those domains.

Please check the SSL certificates that were issued to .tg domains and
that chain up to your root certificates included in Mozilla's program to
ensure that the certificate subscriber actually owns the domains
included in their certificate.

Response Options:

- There are no TLS/SSL certificates issued to .tg domains that chain up
to our root certificates included in Mozilla's program.

- There are TLS/SSL certificates issued to .tg domains that chain up to
our root certificates included in Mozilla's program, but there were no
new validations on .tg domains from October 25 to November 2, 2017.

- There are TLS/SSL certificates issued to .tg domains that chain up to
our root certificates included in Mozilla's program, and we have
re-verified the certificates that were issued to .tg domains from
October 25 to November 2, 2017, and no problems were found.

- We have revoked certificates to .tg domains between October 25 and
November 2, 2017, and have sent information about these revoked
certificates to Mozilla.

- Not Applicable, because our root certificates do not have the Websites
trust bit enabled.

- Other - explain
~~


Thanks,
Kathleen

[Jakob Bohm]

Shouldn't there be an "issued" in there? (as phrased it seems to say
that the revocation, not the issuance, took place during the incident).

> - Not Applicable, because our root certificates do not have the Websites
> trust bit enabled.
>

Wouldn't the .tg incident be equally relevant for the e-mail trust bit?
(In which case the first 3 options should say TLS/SSL/e-mail)

> - Other - explain
> ~~
>
>
> Thanks,
> Kathleen

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Kathleen Wilson]

On 11/13/17 7:22 PM, Jakob Bohm wrote:
>
> Wouldn't the .tg incident be equally relevant for the e-mail trust bit?
> (In which case the first 3 options should say TLS/SSL/e-mail)

Good point. To make it easier, I removed "TLS/SSL", and changed text to
"certificates containing .tg domains".


Updated as follows:

~~

ACTION 8: Check for issuance of certificates containing .tg domains from

October 25 to November 2, 2017.

We believe that the .tg Registry was compromised from October 25 to
November 1, 2017, such that a perpetrator set the Name Server (NS)
Records for some domains to name servers controlled by them, and then

successfully obtained certificates for those domains.

Please check the certificates containing .tg domains that chain up to

your root certificates included in Mozilla's program to ensure that the
certificate subscriber actually owns the domains included in their
certificate.

Response Options:

- There are no certificates containing .tg domains that chain up to our

root certificates included in Mozilla's program.

- There are certificates containing .tg domains that chain up to our

root certificates included in Mozilla's program, but there were no new
validations on .tg domains from October 25 to November 2, 2017.

- There are certificates containing .tg domains that chain up to our

root certificates included in Mozilla's program, and we have re-verified

the certificates that were issued for .tg domains from October 25 to

November 2, 2017, and no problems were found.

- We have revoked certificates containing .tg domains that were issued

between October 25 and November 2, 2017, and have sent information about
these revoked certificates to Mozilla.

[Kathleen Wilson]

On 11/14/17 4:34 AM, douglas...@gmail.com wrote:
>
> Do we believe that this issue has been resolved by the Registry and issuance an resume as normal, or are there ongoing concerns which CAs should be aware of when issuing certificates to .tg domains?
>


Based on information from folks that are monitoring their NS Records, we
believe that the .tg Registry problems were fixed on November 1, and
have remained fixed since then.

I have not looked into how Registries are operated and maintained, so
here is my personal (uneducated) opinion: I think it is possible that
the .tg Registry could be compromised again. I have no idea if all of
the newer Registries are using good network and security protocols,
infrastructure, etc.

I think that we will need to have much deeper investigation and
discussions about Registries, so I have added this to my to-do list, but
I will not be able to get to it until January.

Thanks,
Kathleen

[jo...@letsencrypt.org]

On Tuesday, November 14, 2017 at 8:31:34 AM UTC-8, Kathleen Wilson wrote:
> On 11/14/17 4:34 AM, douglas...@gmail.com wrote:
> >
> > Do we believe that this issue has been resolved by the Registry and issuance an resume as normal, or are there ongoing concerns which CAs should be aware of when issuing certificates to .tg domains?
>
> Based on information from folks that are monitoring their NS Records, we
> believe that the .tg Registry problems were fixed on November 1, and
> have remained fixed since then.

Let's Encrypt disabled issuance to .tg on November 1 as a protective measure. The block remains in place. We'd like to lift the block but we have seen no evidence that the problem was ever acknowledged or fixed by anyone involved in running the .tg registry.

Most of the issuance to .tg during the problematic period was from Let's Encrypt (note that validation was successfully completed, Let's Encrypt did not mis-issue). Since we disabled issuance to .tg on Nov 1 a lack of new suspicious issuance may only reflect our block, not resolution of problems.

The fact that some large companies got control of their domains back may only reflect customer service actions.

We are stuck in a difficult situation where we'd like to re-enable issuance to .tg but we just don't have confidence that the registry is secure. If anyone has any direct evidence we'd greatly appreciate seeing it.

Without more evidence we will simply have to re-enable .tg issuance and monitor it for a period of time.

[Nick Lamb]

On Tuesday, 14 November 2017 16:31:34 UTC, Kathleen Wilson wrote:
> Based on information from folks that are monitoring their NS Records, we
> believe that the .tg Registry problems were fixed on November 1, and
> have remained fixed since then.
>
> I have not looked into how Registries are operated and maintained, so
> here is my personal (uneducated) opinion: I think it is possible that
> the .tg Registry could be compromised again. I have no idea if all of
> the newer Registries are using good network and security protocols,
> infrastructure, etc.

Can we loop in somebody (from ICANN maybe? or the root operators?) who can speak for the top level? Do they actually have any power or influence over ccTLDs at all ? Do these registries in practice actually do anything they're told or are they a law unto themselves?

It seems to me there's a bunch of options here

At one extreme we just accept that some TLDs will be poorly run, entities like Google that have acquired 2LDs in every suffix they can will have cause to regret this but it's not our fight. Certificates for google.tg will be properly issued to whoever happens to persuade the broken .tg registry system to agree they own it that morning, and if asked we point to the .tg registry because it's their problem. Because the DNS is a hierarchy this has no impact for people whose names aren't under poorly run registries, and the incentive to run registries properly lies with the registrars who can expect nobody to bother paying for something that's now effectively worthless.

A middle path is that CA/B or Mozilla on its own, decides that registries which can't manage this sort of thing properly aren't able to deliver on the promise that names should be "meaningful" and so a list of registries will be blacklisted and all names under those suffixes will be ineligible for Web PKI certificates, it would then always be mis-issuance to issue for such names at all.

And at another extreme Mozilla could decide that Firefox, the browser, won't trust such names, and blacklist suffixes at its sole discretion, affected DNS names would simply never get treated as secure in Firefox - it would be acceptable to issue certificates but they won't make any difference for those names.

[Kurt Roeckx]

On 2017-11-15 13:07, Nick Lamb wrote:
> And at another extreme Mozilla could decide that Firefox, the browser, won't trust such names, and blacklist suffixes at its sole discretion, affected DNS names would simply never get treated as secure in Firefox - it would be acceptable to issue certificates but they won't make any difference for those names.

If you want to go for extreme, you could just refuse visit such domains.

But I would instead try to pressure them into fixing things via IANA/ICANN.


Kurt

[jo...@letsencrypt.org]

Let's Encrypt has now received confirmation from CAFE Informatique & Télécom (.tg operators) that the .tg registry was compromised around Nov 1, 2017. Apparently a vulnerability in some front-end software ultimately allowed attackers to access and manipulate the registry database. CAFE Informatique & Télécom believes they have resolved the issues and stated that they are taking steps to further secure their infrastructure.

Let's Encrypt will resume issuance to .tg domains today, with additional monitoring of .tg domain issuance in place for a period of time in order to be confident that the fixes to .tg systems are effective.

[Jeremy Rowley]

We had a conversation with the tg registry, and it looks like the TLD was
compromised until Nov 10. Here's a snippet:

TG Registry (FR): Nous sommes C.A.F.E Informatique & Télécommunications,
gestionnaire technique du .tg. Nous répondons à vos requêtes avec l'accord
de l'ART&P, le gestionnaire administratif du .tg.
TG Registry (EN): We’re C.A.F.E Informatique & Télécommunications, technical
manager of the .tg registry. We respond to your request in agreement with
the Administrative Managers of ART&P.

CAS (FR): On a ete informes qu’il y a quelque jours le bureau
d’enregistrement pour l’extension .tg a eu des problemes de securite.
CAS (EN): We’ve recently been informed that the .tg Registry experienced
some security problems.
TG Registry (FR): En effet, notre plateforme de gestion de noms de domaine a
subi des attaques. Certaines attaques ont eu pour conséquence d'altérer les
informations des noms de domaines appartenant à certains de nos clients.
TG Registry (EN): In effect, our management platform for domain names
suffered an attack. Some attacks did result in the alteration of domain
name ownership records for some of our clients.

CAS (FR): On a besoin de savoir exactement la nature de la probleme que vous
avez eu. Aussi on a besoin de savoir quand le probleme a commence et quand
a ete finalisee.
CAS (EN): What was the exact nature of the problem? Also, we need to know
when the problem started and when it was resolved?
TG Registry (FR) : Nous avons eu une altération des informations des noms de
domaines. Le problème a commencé le 01/11/2017. Il a été réglé et confirmé
comme tel le 10/11/2017.
TG Registry (EN): Alteration of domain name information were made. The
problem started on 1 Nov 2017. We confirmed the problem was resolved 10 Nov
2017.

Hope this helps!

Jeremy

-----Original Message-----
From: dev-security-policy
[mailto:dev-security-policy-bounces+jeremy.rowley=digice...@lists.mozilla
.org] On Behalf Of Kathleen Wilson via dev-security-policy
Sent: Tuesday, November 14, 2017 9:31 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: .tg Certificates Issued by Let's Encrypt

On 11/14/17 4:34 AM, douglas...@gmail.com wrote:
>
> Do we believe that this issue has been resolved by the Registry and
issuance an resume as normal, or are there ongoing concerns which CAs should
be aware of when issuing certificates to .tg domains?
>

Based on information from folks that are monitoring their NS Records, we
believe that the .tg Registry problems were fixed on November 1, and have
remained fixed since then.

I have not looked into how Registries are operated and maintained, so here
is my personal (uneducated) opinion: I think it is possible that the .tg
Registry could be compromised again. I have no idea if all of the newer
Registries are using good network and security protocols, infrastructure,
etc.

I think that we will need to have much deeper investigation and discussions
about Registries, so I have added this to my to-do list, but I will not be
able to get to it until January.

Thanks,
Kathleen

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://clicktime.symantec.com/a/1/mcEC05Cw_1xqHZAtMNn7EcnEg4nah8YZRDPpmi4jv
98=?d=CVUyKj1esC0iPxphVfmlXKmHgqqoVMlZEs52KqNWd2oT-ZKsOrQyh75JHM8fUOmXQvSVYN
Axaf_uL_qw2veBL1uD_NBnuxrVL03BMwmJ5WqSeO1Qyb3EGvXK-WOqcGI5ZqPNUefR37XwxjDImn
gIZ6V-qfcc_hbPRc-1pi89HlCsBAj1fejLAHOId10a-l17fqyWDPPMHNPRUc9gd9k8ulOD9XWv3W
9NzgK45lFwYcWcV9V-hzOhaCpA4j1AHVYrZ3FGrLGHNNhmK4lv4eGuAOzmuGJ19qaFW4rZ_bf-7A
qX7ZLNl5RPblcUQbUJl9jM1pEVgOedqPMkHQTaZfa4YXFkfDfo3Dk8pDlp12_sATizciDKRpPVvY
c7r7_v9B0elPw5r9mqGmYsl7kBhrLgHRBp0xZ3Qwwjoq26GhtpLHE4961NVld1nmZlPAtJ1_svO-
BrEmyoWAd-&u=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-security-polic
y

[Robin Alden]

Hi Kathleen,
Comodo issued a number of certificates to .tg domains during the
period of interest.

We see a history of applications for <something>.gouv.tg certificates which
we had been previously been rejecting and suddenly in the period of interest
we issued them - which might support the notion of the .tg registry being
compromised.
It could, of course, also indicate a sudden burst of activity by the Togo
government in setting up websites. It is hard to tell.

We issued certificates including around 170 names matching
<something>.gouv.tg.
Issued names certificates
19/06/2017 2 1
01/08/2017 1 1
25/10/2017 31 7
26/10/2017 46 15
27/10/2017 7 3
28/10/2017 8 4
30/10/2017 19 8
31/10/2017 20 4
01/11/2017 12 2
02/11/2017 9 3
03/11/2017 8 4
04/11/2017 5 2

and that's when we blocked .tg.

When we first got a heads-up about this we looked at the data and I said
that it looked to me like 25th October was the transition to chaos, since
that is when we issued the first of many gouv.tg certificates.

I hope that helps a little.

Regards
Robin Alden
Comodo CA Ltd

> https://lists.mozilla.org/listinfo/dev-security-policy

[Kathleen Wilson]

Thank you to everyone who has been looking into the .tg Registry problem
and providing valuable information. I greatly appreciate all of your
efforts!

I have updated the related action item in the November CA Communication
to reflect the dates that we believe the .tg Registry was having
problems with NS Records.

~~
ACTION 8: Check for issuance of certificates containing .tg domains from

October 25 to November 11, 2017.

We believe that the .tg Registry was compromised from October 25 to

November 10, 2017, such that a perpetrator set the Name Server (NS)

Records for some domains to name servers controlled by them, and then
successfully obtained certificates for those domains.

Please check the certificates containing .tg domains that chain up to
your root certificates included in Mozilla's program to ensure that the
certificate subscriber actually owns the domains included in their
certificate.

Response Options:
- There are no certificates containing .tg domains that chain up to our
root certificates included in Mozilla's program.

- There are certificates containing .tg domains that chain up to our
root certificates included in Mozilla's program, but there were no new

validations on .tg domains from October 25 to November 11, 2017.

- There are certificates containing .tg domains that chain up to our
root certificates included in Mozilla's program, and we have re-verified
the certificates that were issued for .tg domains from October 25 to

November 11, 2017, and no problems were found.

- We have revoked certificates containing .tg domains that were issued

between October 25 and November 11, 2017, and have sent information