CT2 Log Compromised via Salt Vulnerability

[Jeremy Rowley]

Hey all, 

I'm sad to report that we discovered today that CT Log 2's key used to sign SCTs was compromised last night at 7 pm via the Salt vulnerability (https://threatpost.com/salt-bugs-full-rce-root-cloud-servers/155383/). All other DigiCert CT logs are uneffected as they run on separate infrastructure. We are pulling the log into read-only mode right now.  Although we don't think the key was used to sign SCTs (the attacker doesn't seem to realize that they gained access to the keys and were running other services on the instracture), any SCTs provided from that log after 7pm MST yesterday are suspect. The log should be pulled from the trusted log list.

Happy to answer any questions about what happened, the infrastructure running the other logs, or what remediation we are taking.. 

Jeremy 

[Jeremy Rowley]

Digging into our logs, I think the log should be distrusted for everything after 17:00:02 on May 2. This was the last known good treehead. That head was published at 5:00:00 on Sunday, May 3. All SCTs issued after this don't appear in a tree signed before compromise.

[Matt Palmer]

I'd be extremely interested to know how an attacker managed to make a
network connection to your configuration management system from, presumably,
the public Internet.

- Matt

[Josh Purcell]

I second this. 

[Jeremy Rowley]

Explanation:

Salt works on a pub/sub model. The salt msater publish a job on the even ubs, which the salt minions subscribe to for work. Authentication by the minions is through PKI, which is also used to encrypt all traffic. The salt master for CT log 2 was managing the servers which communicated to the public internet for the purpose of signing and distributing SCTs.  The exploit allows an attacker to connect to the master without authentication and publish jobs to the event bus. At the time CT2 was deployed, digital ocean did not have a private networking feature. It's a very old log (and tangentially one of the reasons we've been trying to shut it down). This means that the saltmaster had to be exposed to the public internet as that was the way salt worked. When the attacker gained access to the event bus, they could have requested the master to deliver the CT server's signing key. We don't think this happened, since the attacker made it clear they were present on the system. Such a request would have also required a knowledge of teh system and how to request such information from the master. That said, we also don't have proof that the attacker didn't request the keys from the master. Therefore, to protect the integrity of CT, it's better to assume that the attacked did request the keys and get them. 

[Jeremy Rowley]

Note that Digicert's CT logs are operated in a separate environment than CA operations. In fact, unique CT logs are operated seperately from other CT logs so the event really is limited to CT2. 

The attack was exactly what is described in the CVE:

CVE-2020-11651An issue was discovered in SaltStack Salt before 2019.2.4 and 3000 before 3000.2. The salt-master process ClearFuncs class does not properly validate method calls. This allows a remote user to access some methods without authentication. These methods can be used to retrieve user tokens from the salt master and/or run arbitrary commands on salt minions.CVE-2020-11652An issue was discovered in SaltStack Salt before 2019.2.4 and 3000 before 3000.2. The salt-master process ClearFuncs class allows access to some methods that improperly sanitize paths. These methods allow arbitrary directory access to authenticated users.

[Devon O'Brien]

Hey Jeremy,

Just to confirm, both of these times are in MST as well?

[Matt Palmer]

On Sun, May 03, 2020 at 07:41:18PM -0700, Jeremy Rowley wrote:
> Salt works on a pub/sub model. The salt msater publish a job on the even
> ubs, which the salt minions subscribe to for work. Authentication by the
> minions is through PKI, which is also used to encrypt all traffic. The salt
> master for CT log 2 was managing the servers which communicated to the
> public internet for the purpose of signing and distributing SCTs. The
> exploit allows an attacker to connect to the master without authentication
> and publish jobs to the event bus.

That requires the attacker to be able to establish a TCP connection (or at
least exchange UDP packets) with the salt master, I assume?

> At the time CT2 was deployed, digital
> ocean did not have a private networking feature. It's a very old log (and
> tangentially one of the reasons we've been trying to shut it down). This
> means that the saltmaster had to be exposed to the public internet as that
> was the way salt worked.

Does the way salt worked require the saltmaster be exposed to the *entire*
public Internet?

> That said, we also don't have proof that the attacker didn't request the
> keys from the master.

That's... unfortunate. Does Salt have the ability to log actions undertaken
from remote, potentially untrusted locations?

I believe you've stated that this CT log does not share any infrastructure
with any other CT logs or DigiCert CA operations. Does that statement
extend to the use of Saltstack itself, and the personnel involved in
designing, deploying, and operating the compromised CT log? That is to say,
is any part of Saltstack present anywhere else within DigiCert's CT log or
CA infrastructure, in any capacity? Has anyone involved in the design,
deployment, or operation of the compromised CT log been involved in the
design, deployment, or operation of any other DigiCert CT log or CA
infrastructure, or are there plans to do so in the future? I include
management personnel in my question, to the degree that they were involved in
reviewing, approving, or overseeing the work of those more directly involved
in the design, deployment, or operation of the compromised CT log.

- Matt

[Jeremy Rowley]

Yes - all times are in MST. 

[Ryan Sleevi]

On Mon, May 4, 2020 at 11:10 AM Matt Palmer <mpa...@hezmatt.org> wrote:

Does that statement
extend to the use of Saltstack itself, and the personnel involved in
designing, deploying, and operating the compromised CT log?  That is to say,
is any part of Saltstack present anywhere else within DigiCert's CT log or
CA infrastructure, in any capacity?  Has anyone involved in the design,
deployment, or operation of the compromised CT log been involved in the
design, deployment, or operation of any other DigiCert CT log or CA
infrastructure, or are there plans to do so in the future?  I include
management personnel in my question, to the degree that they were involved in
reviewing, approving, or overseeing the work of those more directly involved
in the design, deployment, or operation of the compromised CT log.

Hi Matt,

I think it's really important for the health of the community that we look to focus on "blameless" postmortems. I think these questions really uncomfortably hew close to trying to assign blame to individuals, at least as read, and I think that's somewhat counter to a healthy culture of learning and understanding if folks reasonably fear they'll get excorciated for any mistakes. While there are times that it may be appropriate to look at systemic repeat "human error" for which all reasonable training might have accounted for, we're definitely not at that point.

I think it'd be helpful if, rather than the questions about the personnel involved, up to and including management, that you could reframe your concerns in a way that could be addressed in a way that helps the whole ecosystem learn and improve. Understanding your concerns seems much more useful to helping ensure that they're resolved, but it's important that we don't take any failures as the justification for bringing out the tar and pitchforks. If you think I'm overlooking important details that would make these questions salient, it'd be useful if you could provide that context, because right now it feels a bit too direct and hostile for what we'd hope for.

Thanks for understanding!

[Jeremy Rowley]

The master had to be public facing with the way Salt worked when CT2 was set up. The attack exploited the public salt master interface. Salt only started supporting a private interface recently. 

[Pierre Phaneuf]

Time zone is MST or MDT?

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/5c4a92bc-1593-43bf-bbd4-d26332a4eae5%40chromium.org.

[Besmir Zanaj]

Totally agree with Ryan Sleevi's feedback. Does not make any sense to blame something/someone that started the open discussion in the first place. 

We should be looking at best practices, notifying the community, lesson learned and root cause topics instead.

my 2 cents.

Besmir

[Jeremy Rowley]

MST

To unsubscribe from this group and stop receiving emails from it, send an email to ct-p...@chromium.org.

[Matt Palmer]

On Mon, May 04, 2020 at 12:02:55PM -0400, Ryan Sleevi wrote:
> On Mon, May 4, 2020 at 11:10 AM Matt Palmer <mpa...@hezmatt.org> wrote:
> > Does that statement
> > extend to the use of Saltstack itself, and the personnel involved in
> > designing, deploying, and operating the compromised CT log? That is to
> > say,
> > is any part of Saltstack present anywhere else within DigiCert's CT log or
> > CA infrastructure, in any capacity? Has anyone involved in the design,
> > deployment, or operation of the compromised CT log been involved in the
> > design, deployment, or operation of any other DigiCert CT log or CA
> > infrastructure, or are there plans to do so in the future? I include
> > management personnel in my question, to the degree that they were involved
> > in
> > reviewing, approving, or overseeing the work of those more directly
> > involved
> > in the design, deployment, or operation of the compromised CT log.
>

> I think it'd be helpful if, rather than the questions about the
> personnel involved, up to and including management, that you could reframe
> your concerns in a way that could be addressed in a way that helps the
> whole ecosystem learn and improve. Understanding your concerns seems much
> more useful to helping ensure that they're resolved, but it's important
> that we don't take any failures as the justification for bringing out the
> tar and pitchforks. If you think I'm overlooking important details that
> would make these questions salient, it'd be useful if you could provide
> that context, because right now it feels a bit too direct and hostile for
> what we'd hope for.

I'm not looking to blame the people involved, any more than I'm looking to
blame Saltstack. What I'm trying to do is identify any other systems that
need closer attention because their use of software which has been shown to
be deficient, or the fact that they were designed, deployed, operated, or
overseen by people who have been shown to be deficient.

Basically, leaving an internal configuration management platform open to the
entire public Internet is a terrible idea. I could ask about how the
decision to do that was arrived at, but whenever I've asked questions about
decision-making in the past of various CAs, the question has not been
answered, and when I've followed up asking for clarification I've been told
(paraphasing) "it looks like that's all you're going to get". This is a
different approach: since "how did this happen?" (so we can prevent it
happening again) is apparently unanswerable, I'm hoping to identify what
other systems are likely to have suffered similar deficiencies (due to the
same people working on them) so they can be more closely examined.

- Matt

[Alex Gaynor]

(Resending my reply from the group, apologies to those who receive it twice0

> the fact that they were designed, deployed, operated, or overseen by people who have been shown to be deficient.

This is where you go off the rails. This way of approaching these problems isn't compatible with a blameless post-mortem approach, which must focus on the systems themselves, incentives, and considerations which led us here, not on the notion that the engineer(s) who built the system were simply too dumb. This "the problem was with the one person who designed this" approach is the same mistaken approach of the people who say that C/C++ are perfectly safe, it's all these deficient developers who keep putting in UAFs, good devs don't do that. And following this approach leads us away from, not towards, systemic solutions.

Alex

--
You received this message because you are subscribed to a topic in the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/ct-policy/aKNbZuJzwfM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20200505004342.tsntdzivdx3cpjga%40hezmatt.org.

--

All that is necessary for evil to succeed is for good people to do nothing.

[Matt Palmer]

On Mon, May 04, 2020 at 09:51:00AM -0700, Jeremy Rowley wrote:
> The master had to be public facing with the way Salt worked when CT2 was
> set up. The attack exploited the public salt master interface.

I take it that that interface couldn't be selectively firewalled to known
remote endpoints, then. In that case, what risk analysis was performed to
come to the conclusion that leaving that open to the entire public Internet
was acceptable?

- Matt

[Matt Palmer]

On Mon, May 04, 2020 at 08:50:38PM -0400, Alex Gaynor wrote:
> > the fact that they were designed, deployed, operated, or overseen by
> people who have been shown to be deficient.
>
> This is where you go off the rails. This way of approaching these problems
> isn't compatible with a blameless post-mortem approach, which must focus on
> the systems themselves, incentives, and considerations which led us here,

I look forward to reading a detailed explanation of "the systems themselves,
incentives, and considerations" from DigiCert. Until then, there really
isn't anything any of us can contribute.

- Matt

[Devon O'Brien]

Respectfully, this line of questioning is rapidly exhausting its value with respect to gathering information about the incident that's relevant to CT Policy. 

Speaking for Chrome, we appear to have all the data we need to make an informed decision on the matter, which we will be announcing shortly.

-Devon