Review of e-Tugra's Inclusion in Mozilla’s Root Store
Ben Wilson
This email relates to the e-Tugra breach that was described in a blog post by Ian Carroll and subsequent discussions here and in CCADB Public and Bugzilla. We are grateful for the involvement of various individuals and organizations, particularly Ian Carroll and Google Chrome, who have contributed their expertise and resources while investigating this breach.
We are now opening this discussion to help determine whether the e-Tugra root certificates should be removed from Mozilla’s Root Store. We will greatly appreciate your thoughtful and constructive feedback on this.
Below are some questions for us to consider in this discussion.
What were the main concerns raised by the community during the discussions that took place in Bugzilla Bug #1801345? (this is not a complete list; details may be found in the bug)
Mr. Carroll indicated that he was able to log in and conduct reconnaissance on e-Tugra’s email and document storage systems, gaining access to customer PII.
There were security holes in e-Tugra’s internal systems that existed because access to internal resources was not adequately secured, namely, default passwords on some administrative tools had not been changed, and internal applications were left exposed without appropriate access control mechanisms.
Statements by e-Tugra about the lack of impact were refuted by the fact that the contents of 568,647 emails and 251,230 documents were accessible, including access to password resets and confirmation codes.
e-Tugra indicated in their incident report that they would conduct additional vulnerability scanning and penetration testing before the end of 2022. They also said that they were preparing a detailed report of the problem and would provide more information.
None of this information was provided until April 2023.
The explanations in subsequent reports and executive summaries of penetration testing did not fully answer the questions that had been asked.
A detailed analysis was not provided, no root cause analysis was ever provided, and the output of the penetration test (i.e., reports) did not offer any detail addressing concerns raised in the incident report. (The penetration test reports only provided brief statements and insufficient information to evaluate the security posture of e-Tugra.)
Does the benefit of having e-Tugra root certificates in Mozilla’s Root Store outweigh the risk?
The breach that was found by Mr. Carroll may indicate that the CA operators may not have the level of expertise necessary to securely operate a publicly trusted CA.
The slow response by e-Tugra may also be an indication that they may not be prepared to respond to concerns and incidents in the required timely manner.
How would the distrust of e-Tugra affect our users?
Since the e-Tugra root certificates are not currently being used, there would not be any impact to our users.
e-Tugra is not currently issuing TLS certificates within their own CA hierarchy. Instead, e-Tugra has been acting as a reseller for the SSL.com CA, which means that all domain validation and certificate issuance is being performed by systems operated by the SSL.com CA, so SSL.com is the party currently responsible for issuance/security procedures.
If distrusted, could e-Tugra continue to be a reseller for a publicly trusted CA?
Yes, as long as the publicly trusted CA ensures correct systems/procedures/processes in relation to reselling certificates within their CA hierarchy.
If distrusted, could e-Tugra re-apply to have their root certificates included in Mozilla’s Root Store?
If we decide to remove the current e-Tugra roots, then in order for e-Tugra to apply to have their root certificates included in Mozilla’s Root Store again, they would need to:
Implement an infrastructure that is properly secure and tested, with sufficiently detailed artifacts to show that the systems are properly configured.
Then create new root certificates within the properly secured infrastructure.
Then re-apply for inclusion in Mozilla’s root store.
As always, the onus is on the CA operator to demonstrate their ability to operate a publicly trusted CA hierarchy.
How are other root stores addressing this issue?
Google Chrome announced that they will remove the “E-Tugra Global Root CA ECC v3” and “E-Tugra Global Root CA RSA v3” roots from the Chrome Root Store.
Apple stated that the e-Tugra roots are not in the Apple Root Store.
Given that SSL.com has been the operator of the issuing CA, e-Tugra’s breach has not necessitated that we take any urgent safeguarding actions. However, e-Tugra’s problems give us reason to question whether the e-Tugra CA should continue to be in our root store, based on the level of expertise required to securely operate a publicly trusted CA infrastructure. Additionally, e-Tugra’s inadequate responses to the problem and resulting requests for information, give us reason to question e-Tugra’s commitment to the important role that CAs have in ensuring security on the web.
We appreciate your engagement and dedication, and we kindly request that all participants adhere to our Community Participation Guidelines, fostering a respectful and constructive environment for dialogue.
Thanks,
Ben and Kathleen
PS: For your reference, below are excerpts from https://wiki.mozilla.org/CA/Maintenance_and_Enforcement
CA behavior
Did the CA notice the error and take appropriate action in a timely manner?
Was the threat or error properly contained?
Did the CA notice the hacker intrusion and take appropriate action in a timely manner?
Were the CA's defense mechanisms current and sufficient?
Was the intrusion appropriately contained?
Was the CA's behavior reckless?
Did the CA try to cover up the mistake/threat rather than follow Mozilla's Security Policy?
In incident response, Mozilla is looking for the following factors:
A CA takes responsibility for their own actions.
Incidents are taken with an appropriate level of seriousness.
Incidents are handled with urgency.
Root cause analysis is performed.
Any questions posed, by anyone, are answered quickly and in detail.
A reasonably-detailed incident report is made public on what happened, why, and how things have changed to make sure it won’t happen again
Cynthia Revström
I think it is quite clear that e-Tugra has demonstrated that they are
not able to handle the responsibilities that come with being a
publicly trusted CA.
Additionally, as there would be no user impact as a result of
distrusting e-Tugra it seems to me that removal is the obvious choice.
-Cynthia
> You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
> To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaaUPBb1DG4Uc4Cs40K4v56c2hzkdYO-JS8sNN9K%2BzzfYw%40mail.gmail.com.
Kurt Seifried
Web Server / Application Server information disclosure
Outdated versions of certain third-party applications
Cross-Origin Resource Sharing configuration issues
File uploading vulnerability on the support portal
Usage of default error messages
--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaaUPBb1DG4Uc4Cs40K4v56c2hzkdYO-JS8sNN9K%2BzzfYw%40mail.gmail.com.
Israr Ahmed
Dear Mozilla community members,
E-Tugra treated this incident with utmost seriousness upon its report, taking immediate actions as acknowledged by Ian Carroll on November 18, 2022
https://groups.google.com/a/ccadb.org/g/public/c/SXAeHT04TFc/m/AJ8S0XuXAwAJ?utm_medium=email&utm_source=footer
It is important to note that the exploited application did not have any impact on the certificate life cycle process. Specifically, the validation of DV certificates is directly handled by SSL.com, without involving E-Tugra.
In response to the incident, we conducted comprehensive investigations to identify any potential presence of attackers and took the necessary steps to safeguard the integrity of our infrastructure and protect our users' data. Further details regarding the actions taken can be found:
https://bugzilla.mozilla.org/show_bug.cgi?id=1801345#c18
We acknowledge that there were administrative weaknesses in our responsiveness to the community and forums, as well as the completion of the pen testing exercise leading to frustration. However, we want to emphasize that the information provided was fully transparent, and nothing was concealed from the community. We have already shared this information with both Chrome and Mozilla Root programs . While we are willing to share the requested information with the root store, we maintain our stance that there was no exploitation in the certificate life cycle process. Therefore, we kindly request that root stores consider our request to remain listed.
We can confirm that israr....@gmail.com and dto...@gmail.com represent E-Tugra as ah...@e-tugra.com.tr and dto...@e-tugra.com.tr. The duplicate comment was a result of a technical issue where the page did not immediately reflect the posted comment, and it was subsequently posted by another representative, leading to its duplication.
We strongly believe in collaborative work with root stores and community members. We are open to highlighting any information or details that can benefit the community, and we are fully prepared to provide it. We have the necessary evidence, which will be presented to auditors during the upcoming audit (end of July). The audit report will address this incident and related security issues.
Kurt Seifried
Watson Ladd
>
> Dear Mozilla community members,
>
> E-Tugra treated this incident with utmost seriousness upon its report, taking immediate actions as acknowledged by Ian Carroll on November 18, 2022
> https://groups.google.com/a/ccadb.org/g/public/c/SXAeHT04TFc/m/AJ8S0XuXAwAJ?utm_medium=email&utm_source=footer
>
> It is important to note that the exploited application did not have any impact on the certificate life cycle process.
>Specifically, the validation of DV certificates is directly handled by SSL.com, without involving E-Tugra.
> In response to the incident, we conducted comprehensive investigations to identify any potential presence of attackers and took the necessary steps to safeguard the integrity of our infrastructure and protect our users' data. Further details regarding the actions taken can be found:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1801345#c18
>
Is there some other competent security team at E-Tugra that would
respond to a breach of your issuing infrastructure?
>However, we want to emphasize that the information provided was fully transparent, and nothing was concealed from the community.
pages, with plenty of whitespace and thinks "Nov 13, 2022: The issues
were fixed immediately" is an adequate timeline entry. That's after 5
days to write it.
>We have already shared this information with both Chrome and Mozilla Root programs . While we are willing to share the requested information with the root store, we maintain our stance that there was no exploitation in the certificate life cycle process. Therefore, we kindly request that root stores consider our request to remain listed.
>
> We can confirm that israr....@gmail.com and dto...@gmail.com represent E-Tugra as ah...@e-tugra.com.tr and dto...@e-tugra.com.tr. The duplicate comment was a result of a technical issue where the page did not immediately reflect the posted comment, and it was subsequently posted by another representative, leading to its duplication.
>
> We strongly believe in collaborative work with root stores and community members. We are open to highlighting any information or details that can benefit the community, and we are fully prepared to provide it. We have the necessary evidence, which will be presented to auditors during the upcoming audit (end of July). The audit report will address this incident and related security issues.
I think we should distrust e-trust: there is no reason not to.
Sincerely,
Watson Ladd
Israr Ahmed
Dear Kurt Seifried,
Thank you for bringing this to our attention.
We would like to clarify that our both personal and corporate email accounts are subscribed. The comments were initially posted using the Google Groups web interface and at that time, Google Chrome was logged in and synchronized with my personal email.
Moving forward, I will ensure to use my corporate email for all official communications. I understand the importance of maintaining professional correspondence and have already been communicating with root stores using my corporate email.
If you have any further questions or concerns, please feel free to let us know. Thanks.
Ben Wilson
All,
Thanks again to all who contributed your expertise and resources in investigating and commenting on the e-Tugra breach, as described in a blog post by Ian Carroll and in subsequent discussions here on the Mozilla dev-security-policy list, and on the CCADB Public list, and in Bugzilla.
Summary
We have determined that the combination of concerns about e-Tugra, which we previously summarized, indicates that the risk of keeping the current e-Tugra root certificates in Mozilla’s root store outweighs the benefits to our users. Adequate change-control processes were not in place, and non-CA systems with connections to CA systems were not being adequately protected. The security holes that were initially reported were concerning in themselves, and alone they might not have led us to our decision, but these were compounded by e-Tugra’s inadequate responses, as recorded in Bugzilla.
Incident Response
As stated on the Maintenance and Enforcement wiki page, we want to see that:
A CA takes responsibility for their own actions.
Incidents are taken with an appropriate level of seriousness.
Incidents are handled with urgency.
Root cause analysis is performed.
Any questions posed, by anyone, are answered quickly and in detail.
A reasonably-detailed incident report is made public on what happened, why, and how things have changed to make sure it won’t happen again.
e-Tugra’s responses fell short of these expectations.
Incident Report
Our guidance on incident reports (that has recently moved to the CCADB website) states: “The incident report should explain how systems or processes failed, how the mis-issuance or incident was made possible, and why the problem was not detected earlier. In addition to the timeline of responding to and resolving the incident, the incident report should explain how the CA Owner’s systems or processes will be made more robust, and how other CAs may learn from the incident.”
e-Tugra’s incident report (https://bugzilla.mozilla.org/attachment.cgi?id=9330825) was not reasonably detailed, downplayed the seriousness of the incident, and did not provide details on remediation steps. For instance, we requested a timeline of the events that occurred before November 2022, but e-Tugra only responded with “Oct 10, 2022: An Internal application was published in Internet Zone.” The incident report did not contain a comprehensive root cause analysis, and e-Tugra did not explain how things have changed to ensure incidents like these won’t happen again. We expected more detailed information and a stronger sense of responsibility from e-Tugra.
Urgency
e-Tugra’s slow and inconsistent responses during the course of the investigation did not portray appropriate urgency. e-Tugra’s representatives initially said that there were processes in place to conduct penetration testing and vulnerability assessments of its systems. They said that penetration testing would start on November 18, 2022. They also said that information about the penetration test would be shared as soon as it became available. On December 12, we requested that information be provided by January 6, 2023, but it wasn’t until February 2, 2023, (after prompting) that e-Tugra admitted that they were still getting started with the penetration testing company. On February 28 we reminded e-Tugra to provide weekly updates.
The first information on penetration testing was not made available until April 2, 2023, and then the executive summary provided only a few brief statements. The executive summary from a follow-up report also lacked sufficient detail. e-Tugra indicates that everything will be provided to its auditors during its next audit. However, this highlights not only the lack of urgency with which this has been viewed by e-Tugra, but also the lack of information that we have to objectively assess whether meaningful change has taken or will take place at e-Tugra.
Proficiency
e-Tugra’s statements during the course of the investigation were inconsistent and in some cases contradictory. e-Tugra representatives downplayed the severity of the incident by responding, “Only three documents related to some users' (non SSL) agreements are accessed by the reporter” and “No sensitive information disclosure revealed in this issue”. They also said, “no sensitive information is logged in the logging system”. However, these statements were refuted by the fact that the contents of 568,647 emails and 251,230 documents were accessible, including access to password resets and confirmation codes.
e-Tugra representatives first said there was “no connection” with CA systems and then said that there was a "one-way flow". Then they said that their customer portal would allow reissuance for already-validated domain validation certificates, but that the accessed application had no impact on the process. This was contradictory because the password-reset function could have been compromised with information in the log to change a user’s password.
Root Removal
The responses from e-Tugra’s representatives demonstrate that this CA lacks the expertise needed to operate a sufficiently secure environment for a publicly trusted CA. As such, we will remove the following root certificates in our next batch of changes to Mozilla’s root store:
E-Tugra Global Root CA RSA v3
SHA-1 Fingerprint: E9A85D2214521C5BAA0AB4BE246A238AC9BAE2A9
SHA-256 Fingerprint: EF66B0B10A3CDB9F2E3648C76BD2AF18EAD2BFE6F117655E28C4060DA1A3F4C2
E-Tugra Global Root CA ECC v3
SHA-1 Fingerprint: 8A2FAF5753B1B0E6A104EC5B6A69716DF61CE284
SHA-256 Fingerprint: 873F4685FA7F563625252E6D36BCD7F16FC24951F264E47E1B954F4908CDCA13
Note: We will also be removing the expired “e-Tugra Certification Authority” root certificate, as previously planned.
Follow-up
e-Tugra may continue to be a reseller to another CA operator with root certificates in Mozilla’s root store, as long as that other CA operator ensures correct systems, procedures, and processes in relation to reselling certificates within their CA hierarchy, and all domain validation and certificate issuance is performed by that other CA whose root certificates are included in Mozilla’s root store.
e-Tugra may not become an externally-operated subordinate CA chaining up to a root certificate in Mozilla’s root store, and in order for e-Tugra representatives to apply to have new root certificates included in Mozilla’s root store in the future, they would need to:
Provide sufficiently detailed artifacts and testing to demonstrate that they have implemented an infrastructure with processes in place to ensure that systems are rigorously secured and thoroughly hardened.
Create new root certificates within the properly configured/secured infrastructure.
Re-apply for inclusion in Mozilla’s root store, and go through Mozilla’s full root inclusion process.
Thanks,
Ben and Kathleen
Israr Ahmed
Dear Ben, Kathleen, Mozilla Security Policy Team & Community,
We write to express our concern over the proposed decision to remove E-Tuğra Root CA Certificates from Mozilla's Root Store. We have been serving as a trusted certificate authority for the past 15 years, managing not only SSL certificates but also providing qualified signature services. Our rich history and broad experience in the field have allowed us to continuously improve our security measures and maintain a strong commitment to the standards of our industry.
We acknowledge the seriousness of the situation and respect the Mozilla and community's commitment of maintaining the utmost security standards, but we would like to highlight the following points, which we believe are pivotal to the discussion:
1. Immediate Action: As soon as we were alerted about the potential breach, we took immediate action to assess its impact on our certificate authority (CA) system. Our thorough examination confirmed that no certificate was affected as a result of the incident.
2. Timely Disclosure: We notified Mozilla (https://bugzilla.mozilla.org/show_bug.cgi?id=1801345) and the broader community about the incident promptly, in line with our commitment to transparency. We opened the discussion on the Mozilla community forum to ensure that all stakeholders were informed and could contribute to the resolution.
3. Proactive Response: To ensure that our systems were safe, we conducted penetration testing and completed it successfully. The results, which we shared with the community, validated our initial assessment: no certificates were compromised in any way.
4. Ian Caroll's Assumptions: We understand Mr. Ian Caroll's concerns. However, his assumptions that there must have been an impact on our certificates are not substantiated by the facts. We noticed multiple attempts in our logs where Ian Caroll tried to prove his hypothesis. However, no certificate breach was observed, reaffirming our assessment.
5. Language Barriers: We apologize if our responses have not been as prompt or satisfying as expected. We assure you that our priority has always been addressing this issue with complete dedication, but language barriers may have slowed our communication at times.
6. Third-Party Verification: We are in the process of providing all relevant information related to the incident to our auditors for independent verification. This is part of our ongoing commitment to maintaining transparency and trust with the community.
Therefore, we respectfully disagree with the proposed decision to remove E-Tuğra's Root CA certificates from Mozilla's Root Store, as it appears to be based on assumptions rather than confirmed impacts. We kindly ask the community to consider our points above and wait for the independent audit results, which we believe will support our assessment.
We respect the responsibilities delegated to Mozilla as a custodian of public trust and we will continue to work diligently to rebuild confidence in our CA operations.
We appreciate your understanding and your continued trust in E-Tuğra. Thanks
Cynthia Revström
--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/9a6c49f0-9bdf-4775-87c9-19c422466b9fn%40mozilla.org.