Telia CA - incorrect OID value

[pekka.la...@teliasonera.com]

Indident reports:

ERROR IN DV OID VALUE (deviation 4)

How Telia became aware:
Telia got preliminary CA audit report on 25th June 2018. One of its BR deviations was a finding that "17 Telia DV certificates had incorrectly same OID value that was used for Telia OV certificates."

Timeline of actions:
On the same day Telia fixed the OID value into DV profile so that error won't happen again. Telia's opinion is that the incorrect OID value has no impact on any common system but anyway Telia's plan is to revoke all incorrect certificates ASAP and latest at September 2018. Customers need to replace their original incorrect certificate with a new certificate provided by Telia. Telia will update this bug until all incorrect certificates are revoked.

Summary and details of problematic certificates:
About ~300 of Telia DV certificates for a single pilot DV Customer included OV OID 2.23.140.1.2.2 instead of DV OID 2.23.140.1.2.1. All incorrect ones were enrolled between 20-Mar-2018 and 25-Jun-2018. All are logged to CT and can be found using given dates and issuer "Telia Domain Validation SSL CA v1". Certificates are also available in Telia CA database.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now:
Telia CA started to enroll DV SSL certificates in March 2018. Previously all Telia's SSL certificates were OV SSL certificates. The new certificate type was basically sub-type of Telia OV certificate but with fewer subject fields. Its profile was copied from OV and then modified, tested and piloted but still there was this error in the OID value that was undetected because it won't have any effect anywhere and was commonly used by Telia before.

Steps to fix:
1. fix the DV profile; DONE 25-Jun-2018, no errors occurred after that
2. reproduce all incorrect certificates any provide those to Customer; ONGOING, planned to finnish 30-Sep-2018
3. revoke all incorrect ones; ONGOING, planned to finnish 30-Sep-2018
4. Telia CA decided to improve its testing process to avoid similar errors in the future; DONE 6-Jul-2018

[Ryan Sleevi]

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
I think this highlights a rather significant and serious failure on Telia’s
part when establishing a new certificate profile, and I don’t really see
what concrete steps are being taken to remediate this or detect if it has
happened in the past.

1) What process does Telia have in place for the review of profiles before
being deployed to production?
a) In light of this, how has that process changed?

2) What level of training is provided to those employees tasked with
reviewing?
a) Frequency of training
b) Frequency of reevaluating the training materials?
c) Independent assessments of those training materials?

3) These certificates materially misstate the level of validation provided
by Telia, and the BRs require revocation within 24 hours. What steps has
Telia taken to ensure this?

I cannot understate how significant this failure is. At least one CA
(Turktrust) through a failure of process to adequately review, configure,
and test their certificate profiles, ended up releasing CA certificates
into the wild to entities not qualified to receive them. The result of that
ultimately ended with them exiting the CA business after finding it
difficult to get their products universally trusted.

While it may be argued by some that this is less severe, because it was
“just” an OID, the demonstrated failure of that process calls into
fundamental question the operations of Telia, and the confidence that a
similar incident won’t occur.

Part of the reason for these postmortems is to ensure that all CAs can
learn and improve their best practices, through the mistakes others have
made, so that the public can reliably trust such certificates. As such,
there is an onus on Telia to demonstrate how this is different from past
incidents, how they have learned and incorporated changes from those past
incidents into their current processes, and where those improvements were
deficient.

While it sounds as if Telia did not take those lessons to heart when they
occurred, as the CA community was still unaccustomed to learning from
others mistakes, I think we as a community need to understand - in concrete
and thorough detail - what the old processes were, why they failed, and how
they’re being improved. My questions are merely an illustrative attempt to
show some of how that might be done, but it is by no means exhaustive.

[Lewis Resmond]

Am Freitag, 3. August 2018 10:51:34 UTC+2 schrieb pekka.la...@teliasonera.com:
> Steps to fix:
> 1. fix the DV profile; DONE 25-Jun-2018, no errors occurred after that
> 2. reproduce all incorrect certificates any provide those to Customer; ONGOING, planned to finnish 30-Sep-2018
> 3. revoke all incorrect ones; ONGOING, planned to finnish 30-Sep-2018
> 4. Telia CA decided to improve its testing process to avoid similar errors in the future; DONE 6-Jul-2018

Why will the detecting/revocation process take 2 months? I think this is an extremely long time period for misissued certificates (I personally think a wrong OID is a misissurance). Also, shouldn't it be quite easy to find all wrong certificates with something like an script or query, or does it require a lot of human intervention?

[pekka.la...@teliasonera.com]

Telia got a serious lesson with this incident that should not have happened. Important detail also to know is that certificates were not issued to wrong entities and issuing new certificates with wrong OID field was prevented immediately.

1) Telia has a development process with multiple steps when doing a change to SSL process. Some steps of the process include creating test certificates in test and pre-production systems with documented change plan and a review. Unfortunately test certificates were using test OID values so that the problem couldn’t be detected at test side. Telia has analysed reasons that caused this error. The main reason was not adequately implemented testing. Test process didn't include certificate comparison correctly against so-called model certificate. Telia has model certificates for each certificate type that are used in comparison when any certificate profile changes. This time there wasn't DV model certificate at all (except in test system with test OID) because DV was a completely new certificate type for Telia. OV model certificate (that had OV OID value) was used instead by the reviewers. Telia should have created a DV model certificate at first. In model certificate creation there are several eye pairs including senior developers when accepting a new one. As a resolution Telia has now enhanced processes so that it is mandatory to create model certificate when completely new certificate type is created.
2) We have concluded that the main reason for this problem was not a lack of training but the incomplete test process and documentation. CA audits have annually evaluated Telia training. Recommendations about improvements have been documented into our internal audit reports if necessary. Recommendations (or issues) from CA auditors are always added to Telia Security Plan to improve Telia CA process continuously. Persons involved in the review have got many different types of training that vary from general security to deeper CA software related trainings. E.g. recently Feisty Duck – The Best TLS training in the World and several trainings from our CA Vendor.
a) CA software vendor trainings have been held quarterly.
b) Vendor keep the materials up to date and we update our own training materials annually or when needed
c) CA audits have annually evaluated Telia internal trainings to Registration officers

3) When this problem was discovered the issue was immediately escalated according to Telia Incident process. One of the main steps of Telia incident process is to evaluate the effect of the incident. This time the evaluation result was that no immediate risk was caused as OID was correct OV OID, all certificates with wrong OID fields were issued to known Telia hosted service customers, even though the issue itself was confirmed serious. Telia prevented the issue to repeat by fixing the profile on the same day. As none of these certificates were issued to wrong entities Telia decided to do the replacement/revoking in the organized way that overall harm would be minimal by replacing these certificates with the customers.


Telia will revoke all incorrect certificates. Most of them have already been revoked. Today ~100 is left to be revoked in co-operation with the Customer.

[Ryan Sleevi]

Thanks! I think this is more in line with the goal of these discussions -
trying to learn, share, and disseminate best practices.

Here, the best practice is that, prior to any configuration, the CA should
determine what the 'model' certificate should look like. This model
certificate is, in effect, the technical equivalent of their certificate
profile (e.g. 7.1 of a CA's CP/CPS) - indeed, it might even make sense for
CAs to include their 'model certificates' as appendices in their CP/CPS,
which helps ensure that the CP/CPS is updated whenever the profile is
updated, but also ensures there's a technically verifiable examination of
the profile.

Going further, it might make sense for CAs to share their model
certificates in advance, for community review and evaluation - although
we're not quite there yet, it could potentially help identify or mitigate
issues before hand, as well as help the CA ensure it's considered
everything in the profile.

Similarly, examining these model certificates through linting is another
thing to consider, and to compare these against the test certificates'
linted results. One thing to consider with model certificates is every
configurable option you allow (e.g. key size) can create different model
certificates, so as a testing procedure, you'll want to make sure you have
model certificates for every configurable option, as well as consider test
certificates for various permutations. For example, lets say you're
introducing a new subject attribute to a certificate - as part of
developing your model certificate, and your test certificate, you'll likely
want to examine the various constraints on that field (e.g. length of
field, acceptable characters), and run tests to make sure they produce the
correct and expected results. Consider situations like "all whitespace" -
does it do the expected thing (which could be to omit the field and allow
issuance, or prevent issuance, etc).

As far as training goes, it does sound like there is an opportunity for
routine training regarding changes to the BRs (and relevant RFCs), to make
sure the team constructing and reviewing profiles know what is and is not
acceptable. While it's good to examine the policies for RAs, looking more
holistically, you want to make sure the team tasked with creating and
reviewing these models is given adequate support and training to know - and
critically evaluate - what is or isn't permitted.