Re: Security concerns with the e-Tugra certificate authority

[Kurt Seifried]

Normally I would say e-Tugra needs to reissue all their certificates or like in this case; it would appear they need to reestablish that the certificates were issued properly, which means having all their customers re-create them, establish domain validation, etc. 

But in this case, I think it's so severe that they should be removed and be made to re-apply to ensure all their security controls/processes are up to standard because they clearly are not. This isn't a little mistake. this shows a massive failure at all levels, technically and business-wise (not removing default passwords, seriously... that's literally the most basic of the basic, what else did they get wrong?).

Additionally, did an attacker possibly get a whole set of new certificates over the summer for their domain names (just a few days ago, etugra renewed everything using their own CA, but they also did a lot of them in July/August of this year).

https://crt.sh/?q=e-tugra

https://crt.sh/?q=etugratest.com

Also, should this discussion be moved to https://groups.google.com/a/ccadb.org/g/public ? Added pub...@ccadb.org.

On Thu, Nov 17, 2022 at 10:26 AM Ian Carroll <i...@ian.sh> wrote:

Hi there,

Today I published a blog post at https://ian.sh/etugra, describing several serious security issues I discovered in the e-Tugra certificate authority. I was able to obtain access to two e-Tugra administrative systems using default passwords, which disclosed numerous amounts of subscriber PII and verification details, and appeared to impact e-Tugra's domain control validation processes.

I am concerned that it is possible for these trivial vulnerabilities to be present in a publicly-trusted certificate authority. In light of the recent Symantec news, certificate authorities are clearly being targeted by nation states, but these vulnerabilities could have been discovered by any amateur security researcher. From what I have seen, I firmly believe that additional security issues likely exist in e-Tugra's infrastructure, and they may already be known to adversaries.

The Network and Certificate System Security Requirements require an annual penetration test, or whenever the CA believes there are material changes. Based on this issue, I am concerned that this control is not sufficient to protect certificate authorities against application security issues, and I am concerned that e-Tugra is not following this control. I am also concerned with the lack of vulnerability disclosure programs and bug bounty programs that are operated by CAs in general; indeed no certificate authority at all appears to run a bug bounty program at the moment.

I would suggest that e-Tugra be compelled to take remedial actions such as performing a comprehensive penetration test on their external infrastructure, and building processes to ensure that future applications that they deploy are secure. I also believe e-Tugra should ensure that these issues did not have the ability to compromise domain-control validation for any certificates still valid today.

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/37cff300-b57a-4b38-82e0-a514b4557b07n%40mozilla.org.

--

Kurt Seifried (He/Him)
ku...@seifried.org

[Kurt Seifried]

On Thu, Nov 17, 2022 at 2:59 PM ke ju <meni...@gmail.com> wrote:

I agree that something should be done swiftly. If you did this, who else could have. Also, has the authority been notified, and have they simply changed the passwords yet? Ie could a threat actor watching this list now go there and get into the system after the disclosure

To be clear, I'm not the original reporter. They also posted to the old list which is why I cross-posted it. 

 

Their blog entry has a timeline:

• Nov 13, 2022 4:10: Initial contact to e-Tugra about administrative systems
• Unknown: Administrative systems no longer reachable on the internet
• Nov 13, 2022 18:50: Second set of issues reported to e-Tugra, follow-up on initial issues that appeared fixed
• Nov 14, 2022 8:35: Initial reply from e-Tugra saying they are working on resolution
• Nov 14, 2022 17:18: Asked how to report security issues in the future
• Nov 16, 2022 22:52: Notified e-Tugra of impending disclosure
• Nov 17, 2022: Disclosed this post

[Ian Carroll]

The major issues mentioned in the post are resolved after I notified e-Tugra of them. 

It is important to note that I did not perform an extensive amount of testing of e-Tugra’s systems, given I do not speak Turkish and I found enough issues as it is. Until e-Tugra undertakes a comprehensive security test of their systems, I would assume there are still other vulnerabilities present somewhere else.

--

Thanks,
Ian Carroll

[ke ju]

I agree that something should be done swiftly. If you did this, who else could have. Also, has the authority been notified, and have they simply changed the passwords yet? Ie could a threat actor watching this list now go there and get into the system after the disclosure

On Thursday, November 17, 2022 at 12:52:37 PM UTC-5 ku...@seifried.org wrote:

[Israr Ahmed]

Hi,

We really appreciate the efforts of the reporter in identifying some security issues. At the same time, we are also glad that our customer facing SSL ecosystem is secure and hence not vulnerable to security threats.

It is our internal ecosystem, deploying some other applications than our CA /SSL certificate life cycle system. Right after receiving the email, we made an immediate action to resolve the default password problem. Both SSL and other products have their own separate ecosystems and processes which do not conflict with each other. The logs and email access you got was related to resellers, customers and application processes but not for CA SSL certificate issuance. As claimed in the blog, he was unable to access the DV SSL validation email because a separate CA email validation component is in operation and not accessible to any system. 

eTugra CA/SSL ecosystem is pen-tested annually and all the infrastructure changes ensuring the CA/B Network Security Guidelines are in place. We once again want to appreciate you attempting to point out any possible threats as such reports always help reassess our system and come back with extra feel of security.

@Kurt Seifried, on your point (just a few days ago, etugra renewed everything using their own CA, but they also did a lot of them in July/August of this year

We were in the process of inclusion of new Root CAs. We generated both test sub-CAs and test SSL certificates. You can find more details from this link:
https://bugzilla.mozilla.org/show_bug.cgi?id=1628720#c31

An incident report is also published on this link:

https://bugzilla.mozilla.org/show_bug.cgi?id=1801345

Regards,

Ahmed

[Davut Tokgoz]

Hi,

We really appreciate the efforts of the reporter in identifying some security issues. At the same time, we are also glad that our customer facing SSL ecosystem is secure and hence not vulnerable to security threats.

It is our internal ecosystem, deploying some other applications than our CA /SSL certificate life cycle system. Right after receiving the email, we made an immediate action to resolve the default password problem. Both SSL and other products have their own separate ecosystems and processes which do not conflict with each other. The logs and email access you got was related to resellers, customers and application processes but not for CA SSL certificate issuance. As claimed in the blog, he was unable to access the DV SSL validation email because a separate CA email validation component is in operation and not accessible to any system. 

eTugra CA/SSL ecosystem is pen-tested annually and all the infrastructure changes ensuring the CA/B Network Security Guidelines are in place. We once again want to appreciate you attempting to point out any possible threats as such reports always help reassess our system and come back with extra feel of security.

@Kurt Seifried, on your point (just a few days ago, etugra renewed everything using their own CA, but they also did a lot of them in July/August of this year

We were in the process of inclusion of new Root CAs. We generated both test sub-CAs and test SSL certificates. You can find more details from this link:
https://bugzilla.mozilla.org/show_bug.cgi?id=1628720#c31

An incident report is also published on this link:

https://bugzilla.mozilla.org/show_bug.cgi?id=1801345

[Ryan Hurst]

In my personal capacity, as I review the published facts that exist here, there are a few broad things from a BR perspective that stand out to me:

• All CAs are required to have incident response and compromise handling procedures. The timeline presented in Ian’s post suggests that no such plan exists, the plan that exists is insufficient, or the plan is not staffed.

• All CAs are required to perform an annual Risk Assessment. The existence of administrative consoles on the internet, default administrative passwords being in use, and the ability to in an uncontrolled fashion add new administrative users tells us this either is not happening or is being performed by individuals with zero understanding of security.

• All CAs are required to perform an annual penetration test. The existence of these trivial issues suggests that either this was not happening or it was being performed by individuals with zero understanding of security.

While PKI-related auditors tend to have a limited understanding of PKI and security, in all my time in this industry they have always demonstrated an understanding of IT accounting principles that would be sufficient to identify these trivial deficiencies.

I would go so far as to say that these are such basic issues the fact that e-Turga has continually been able to acquire a sufficiently clean audit report to stay within the various programs suggests either a disqualifying skill deficiency or an integrity issue related to the auditing firms.

I also think this incident shows that the current root program requirements and BRs are too generous when it comes to trusting CAs to implement reasonable incident response procedures.

Some things that should be considered moving forward include:

1. Requiring the availability of a documented method for reporting security concerns.

2. Requiring a minimum response time for the initial response to these reports.

3. Requiring that the final disposition of these reports be communicated to the reporter.

4. Requiring that subscribers be notified of any compromises to the certificate management-related systems.

5. Today only issues related to a security issue requiring revocation of an intermediate or cert chaining to a trusted root require the creation of a bug in Bugzilla, this should be expanded to minimally include compromise of any certificate management-related system.

6. Requiring that all security incidents be summarized in some reasonable detail and made available in CCADB for root program administrators.

7. Requiring the identification of who performed the penetration testing, a summary of what was assessed, and a summary fo their findings.

I also believe a common theme in CA failures like these is the lack of qualifications and independence of the auditors and security professionals used to assess the correctness and conformance of the systems. 

I don’t know how best to address this but this incident also seems to suggest that the requirement to use a “qualified auditor” and penetration testers with the “skills, tools, proficiency, code of ethics, and independence” is not sufficient.

Ryan Hurst

(Personal Capacity)

[Ian Carroll]

In the E-Tugra incident report, they state:

> Only three documents related to some users' (non SSL) agreements are accessed by the reporter.

I just want to clarify that this issue impacted 251,230 uploaded user documents, as shown in their administrative panel, as well as over 500,000 sent emails. I may have only accessed three documents during my testing, but the scope of this issue was much larger.

The incident report states:

> Customers sensitive information e.g. credentials and similar information could not be obtained and are in safe condition.

However, E-Tugra password resets, verification codes, and login codes for SMS were all visible in this portal. Explicit passwords may not have been visible, but many E-Tugra users likely normally log in via SMS, which were impacted by this. Additionally, there were customer portal issues not disclosed in my post and I am unsure if these have been investigated for abuse.

The incident report does not show a sufficient level of investigation (or just does not detail enough of it), in my opinion. There's no information on when this issue was introduced and how long they may have been exposed to this issue. There is a statement that activities were reviewed over the past year, but were the issues present for longer than this? A logging system was one of the two impacted systems as well; could logs for this have been tampered with/modified? The customer portal is also not addressed.

--

Thanks,
Ian Carroll

[Israr Ahmed]

Hi Ian Carroll,

We are working with several internal teams to get their feedback and analysis on the reported problems.

We are compiling a detailed report based on feedback from each team before we publish it. We hope that we will able to complete this activity within a week time.

[Kurt Seifried]

Two simple questions that desperately need answers:

1) Does E-Tugra have specific controls in place to ensure systems deployed to production are secured (e.g. default accounts disabled or changed)?

2) If #1 is sufficient, how and why did they fail? If #1 is not sufficient why is it missing such fundamental controls? 

3) Has E-Tugra validated ALL other existing servers/services (both public facing and internal facing) to ensure similar lapses haven't occurred? 

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/cc07165b-31bd-4e2d-a96b-e4b6d9611445n%40mozilla.org.

[Ian Carroll]

E-Tugra posted a decent reply to the incident report at https://bugzilla.mozilla.org/show_bug.cgi?id=1801345. I don't have a Bugzilla account and think it would be more appropriate to respond to it here. In general, the timeline they have shared is inspiring, and it is good that they took swift action on my report. They didn't really communicate with me about it during the incident, but it is good to see that it was taken seriously internally.

However, I am still concerned with what E-Tugra considers logical separation from their CA infrastructure. The administrative tools were clearly connected to EJBCA, as the logging tool had explicit references to calling the EJBCA APIs. Additionally, the E-Tugra panel at panel.e-tugra.com.tr had its authentication system compromised due to the ability to see all emails sent from this. How does this website to buy certificates, used by many customers, have logical separation from the CA system?

There are many statements about M of N, HSM access, etc which do not appear to be relevant to this issue. Unless E-Tugra is hand-copying purchased certificates into an air-gapped issuance infrastructure, the site to buy certificates will have some degree of connection to the issuance infrastructure. It would be useful to enumerate the specific details here instead of making blanket assertions about logical separation which appear to be wrong.

[Peter Gutmann]

Ian Carroll <i...@ian.sh> writes:

>There are many statements about M of N, HSM access, etc which do not appear
>to be relevant to this issue.

That's not specific to e-Tughra though, that's standard for CAs where what
gets audited is all the fancy security mechanisms around the CA's private
key(s) and what barely, or not at all, gets audited is the various RAs that
pull the CA's strings.

Years ago I saw a cartoon lampooning a certain country's defence policy which
had lifeguard-style flags set up on a piece of open ground and a sign between
them saying "Please attack between the flags". With CA's it'd be "please
audit between the flags".

Not defending or criticising e-Tughra, just pointing out that this isn't their
fault, it's How CAs Are Done.

Peter.

[Kurt Seifried]

On Sun, Feb 26, 2023 at 2:22 AM Ryan Hurst <ryan....@gmail.com> wrote:

This thread and associated bug have been silent for an uncharacteristically long time, and I am curious as to when this issue will be closed.

Furthermore, I would like to understand what changes will be put into place to clarify appropriate incident handling behavior. It is important that Mozilla establishes a clear protocol for handling security incidents and communicates this effectively to all participants.

I am also curious in how Mozilla will choose to interpret the facts that have been made available. The way in which this incident is handled will establish a precedent for future security incidents, and it is important that Mozilla approaches this with a clear and consistent stance.

Ryan Hurst

So my take on this is: we're still in the 2000 era of "apply to be a root ca, get in by default, stay in by default", that is to say, if you have all the paperwork and don't do something terrible, well you're good to go. Witness:

SERPRO issued numerous malformed/bad certificates indicating a lack of the most basic controls and voluntarily withdrew their application and can re-apply later. 

BJCA.cn has links to what appears to be spyware, but has said "it's fine" and they have been approved:

This is notice that I am closing public discussion and that I am recommending that we approve BJCA’s inclusion request.
This begins a 7-day "last call" period for any final objections.  

Also no mention or suggestion of limiting them to e.g. .cn for example. 

TRUSTCOR not only crossed some lines but then publicly made statements that were later found to be false, I'm guessing had they "fallen on their sword" and apologized they'd still be a CA.

Mailing list participation is another good indicator that nobody really cares, the same 20-30 people are posting, on issues that affect the entire world. 

The reality is that nobody really cares, nothing that bad has happened (at least in the western world, ignoring the spyware and dead journalists, and repression in various countries). I have a briefing on this and it boils down to "if you want to be especially paranoid do what VISA does (https://developer.visa.com/pages/trusted_certifying_authorities), there's no point in trying to prevent bad CAs from getting in or staying in".

[Ryan Dickson]

All,

We’d like to extend our appreciation to Ian Carroll for reporting this issue to us, and for Ian’s continued availability during the incident’s discussion (both here and on Bugzilla).

After full consideration of the available information related to the vulnerabilities disclosed at https://ian.sh/etugra, including that of incident reports and other public responses, we have decided that in order to appropriately protect and safeguard Chrome users, the following e-Tugra Root CA certificates will be removed from the Chrome Root Store: 

1. E-Tugra Global Root CA ECC v3 

2. E-Tugra Global Root CA RSA v3

Most Chrome users will automatically receive these updates via Component Updater immediately after publication of Version 11 of the Chrome Root Store (expected no later than June 15, 2023). In the event that a Chrome user has disabled Component Updater, these changes will be enabled by default beginning in Chrome Version 115 (stable release approximately July 18, 2023). No additional Chrome user action is required.

We only observe test certificates chaining to the roots listed above (examples [1], [2], and [3]). Consequently, it is our understanding that there is no impact to e-Tugra subscribers or Chrome users as a result of this change. 

We may take further action, or accelerate the timeline described above, as additional information becomes available to us.

- Ryan

On behalf of the Chrome Root Program

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CABqVa3-JgfmFu_9H4Bic6eC2ZnKA8wSG5aewyUH06CC49CjSkA%40mail.gmail.com.

[Kurt Seifried]

I'm curious, can we get any information from the other major browser vendors as to whether or not they are removing these certificates, and if not why not? Thanks!

[Clint Wilson]

Hi Kurt,

FWIW, these Root CA certificates have not been accepted into the Apple Root Program.

Cheers,

-Clint

You received this message because you are subscribed to the Google Groups "CCADB Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@ccadb.org.
To view this discussion on the web visit https://groups.google.com/a/ccadb.org/d/msgid/public/CABqVa39akV6CjPTfBr8oKumv6Q%2B5Tw8%2BWCME38-fhQLx7C%3DHJQ%40mail.gmail.com.

[Kathleen Wilson]

Regarding Mozilla's root program, we will start a discussion about this in MDSP soon.

Thanks,

Kathleen