Security breach at multiple Federal agencies via SolarWinds
2 views
Skip to first unread message
goossbears
Dec 17, 2020, 1:32:22 PM12/17/20
to BerkeleyLUG
Since it's been reported for a few days now *without* the mass posting of counter-warnings such as "False News of Zero Day! False News of Zero Day!", I'd therefore guess that this news of the security breach was (and possibly still is) likely valid.
E.g., from at least ...
- Gizmodo's 'Feds Still Trying to Determine How Screwed They Are After Massive SolarWinds Hack', https://gizmodo.com/feds-still-trying-to-determine-how-screwed-they-are-aft-1845888076
- CNN's Politics' article 'Massive hack of US government launches search for answers as Russia named top suspect', https://www.cnn.com/2020/12/16/politics/us-government-agencies-hack-uncertainty/index.html
- CNN's Business article 'Why the US government hack is literally keeping security experts awake at night', https://www.cnn.com/2020/12/16/tech/solarwinds-orion-hack-explained/index.html
- The BBC News' Tech article 'SolarWinds Orion: More US government agencies hacked', https://www.bbc.com/news/technology-55318815
- The BBC News' Tech article 'SolarWinds: Why the Sunburst hack is so serious', https://www.bbc.com/news/technology-55321643
Further thoughts and insights on this from Michael P, Rick M, Thomas L, and anyone else here?
-A
Rick Moen
Dec 17, 2020, 2:15:42 PM12/17/20
to BerkeleyLUG
Quoting goossbears (acoh...@gmail.com):
> Further thoughts and insights on this from Michael P, Rick M, Thomas L, and
> anyone else here?
I'll pass on my late-night posting to CABAL's mailing list. As an
> Further thoughts and insights on this from Michael P, Rick M, Thomas L, and
> anyone else here?
additional comment, cybersecurity firm FireEye, cited below as one of
the victims of the software-chain infiltration, i.e., one of
SolarWinds's customers who bought and ran the trojaned Orion Platform
network-management software, was also a _key good guy_. FireEye
figured out that their retail copies of Orion were up to no good (had
briefly breached FireEye corporate security from inside the firm's own
networks) and alerted Department of Homeland Security (and alerted
SolarWinds).
https://www.bloomberg.com/news/articles/2020-12-15/fireeye-stumbled-across-solarwinds-breach-while-probing-own-hack
My point is that Texas-based proprietary software company SolarWinds, Inc.
had been utterly clueless about having had their entire software
production chain taken over for months, and had to be informed of their
stunning incompetence and its catastrophic effects by a customer.
The phrase 'You had _one_ job!' comes to mind.
https://www.youtube.com/watch?v=zHCzlCoDBCI
One obvious lesson for Linux users is that it's a reminder that blithely
running some chump corporation's proprietary software exposes you to
risks that you would avoid if you said 'I'll pass' -- and that
code-signing can be just another way to go wrong with confidence...
as three million users of Google Chrome and Microsoft Edge are finding
out:
https://arstechnica.com/information-technology/2020/12/up-to-3-million-devices-infected-by-malware-laced-chrome-and-edge-add-ons/
"How could the Vimeo Video Downloader extension have been unsafe? It was
signed by the [Google|Microsoft] online store!"
I suspect I'll write about the latter story on CABAL's mailing list.
Date: Tue, 15 Dec 2020 22:52:30 -0800
From: Rick Moen <ri...@linuxmafia.com>
To: cons...@linuxmafia.com
Subject: Security breach @ multiple Federal agencies via SolarWinds Orion software
Organization: If you lived here, you'd be $HOME already.
In this posting, I'll be trying in real time to figure out the
substantive reality behind a current news story. Example:
https://gizmodo.com/feds-still-trying-to-determine-how-screwed-they-are-aft-1845888076
Headline is:
Feds Still Trying to Determine How Screwed They Are After Massive SolarWinds Hack
by Tom McKay
RM: There are recurring problems with IT press coverage of security
items, especially security breaches. 1. Where, as is frequently the
case, somebody messed up, the details go underreported because the
people who know don't want to talk about it. 2. IT reporters usually
don't understand security very well, and tend to uncritically crib from
press releases.
A cyberattack that began by targeting an IT firm used by numerous
federal government agencies, Fortune 500 companies, and other high-value
targets is shaping up to be a historic event.
The U.S. government is still reeling after the detection of a massive
foreign intrusion into federal computer systems at agencies including—at
a minimum—the Department of Homeland Security, the Treasury, and the
Commerce Department; [...]
Those responsible built a backdoor into Orion, an IT management
software produced by SolarWinds, possibly by breaking into Microsoft
email accounts and other systems, according to the Wall Street Journal
[link]. They then used it to contaminate software updates provided by
the company with malware in March and June 2020.
To unpack that: Private US company SolarWinds, Inc. publishes
proprietary MS-Windows software for businesses to help manage their
networks, systems, and information technology infrastructure. For
obvious reasons, any such software is itself security-sensitive and runs
with elevated privilege. In Spring 2020, the Russian Federation Foreign
Intelligence Service ('SVR'), specifically its APT29 aka Cozy Bear team,
managed to break into the crown jewels at SolarWinds, Inc., gaining
control of a software signing key for the production software chain,
which was then used to gain 'tokens' for other highly privileged roles
at SolarWinds, and among other thing insert remote-backdoor software
into a binary software library (SolarWinds.Orion.Core.BusinessLayer.dll)
used in future releases of SolarWinds's Orion Platform software product.
So, the 'malicious' code in question then went out signed by
SolarWinds's release-code key, and so went out automatically to
customers as supposedly authentic code.
This root-level compromise of a piece of widely used commercial
off-the-shelf (COTS) software snagged _lots_ of victims. Those who've
admitted getting suckered include:
o NATO
o US Treasury Dept.
o US Commerce Dept. National Telecommunications & Information Administration
o US Dept. of Homeland Security
o EU Parliament
o UK Health Service
o UK Home Office
o cybersecurity firm FireEye (!)
o pharmaceutical and biopharmaceutical company AstraZeneca (probably)
How did SolarWinds, Inc. get H4X0Red? Maybe, by being really
mind-bogglingly stupid?
https://www.msn.com/en-us/news/politics/notorious-hacker-fxmsp-sold-access-to-solarwinds-machines-report/ar-BB1bXZwj
[...]
Vinoth Kumar, a security researcher, told the outlet that he warned
SolarWinds that their update server could have been accessed by "any
attacker" with ease last year because the password was set to
"solarwinds123." Kumar first notified the company of the issue on
November 19, 2019 and the company responded three days later, according
to emails he supplied to Newsweek.
Kumar believes the vulnerability may have been present as far back as
June 2018.
[...]
Or maybe not?
The recent breach, allegedly by Russian hackers, is also unlikely to
be directly related to the password vulnerability since it took place
months after the issue was remedied.
Doesn't seem reassuring, anyway.
SolarWinds, Inc. asks customers to un-fsck themselves as follows:
SolarWinds asks customers currently using Orion Platform v2020.2 with
no hotfix installed or 2020.2 HF 1 to upgrade to Orion Platform version
2020.2.1 HF 2 as soon as possible to ensure the security of your
environment.
https://www.solarwinds.com/securityadvisory/faq
If I were a customer, I'd want the answer to the question 'What
happened, guys, and why should I feel reassured that it cannot ever
happen again?' Is that addressed in their security advisory FAQ, you
ask?
Why didn’t SolarWinds catch this vulnerability before it happened?
This attack was very complex and sophisticated. The vulnerability was
crafted to evade detection and only run when detection was unlikely.
Um, guys?
How do you know the new build is secure?
We have limited access rights to our build environment to only those
necessary and added additional controls to limit access further. As an
added precaution, we are using a new code signing certificate for our
new builds.
Um, _guys_? Why did this fail the first time?
With these processes in place how was your code compromised?
We are not aware that the SolarWinds code base was compromised.[...]
You're kidding.
[...]Our initial investigations point to an issue in the supply chain
resulting in a compromise of our product that inserted a vulnerability
within its Orion monitoring products which, if present and activated,
could potentially allow an attacker to compromise the server on which
the Orion products run.
'An issue in the supply chain'?
Here's the thing: If you do code-signing competently, you can no longer
pass the buck to 'the supply chain', because any (hypothetical)
tampering downstream from your crown-jewels signing machine would result
in the modified software no longer validating as signed by the signing
key of record.
So, the logical inference is that the above is poppycock, that
SolarWinds's code-signing infrastructure, the crown jewels, was indeed
compromised. And, by implication, SolarWinds, Inc. is either in denial
about this fact and is delusional, or is clumsily lying. The latter
interpretation would be a little more reassuring than the former, IMO.
Let's see what CSO Online says:
https://www.csoonline.com/article/3601508/solarwinds-supply-chain-attack-explained-why-organizations-were-not-prepared.html
SolarWinds stated that its customers included 425 of the US Fortune
500, the top ten US telecommunications companies, the top five US
accounting firms, all branches of the US Military, the Pentagon, the
State Department, as well as hundreds of universities and colleges
worldwide.
The SolarWinds software supply chain attack also allowed hackers to
access the network of US cybersecurity firm FireEye [...]
/me reads many more paragraphs.
Nope, no useful insights from CSO Online.
Brian Krebs (https://krebsonsecurity.com/) has started to cover the
story, but in fairness it's quite new. (I expect he will have useful
things to say, and recommend his site.)
There's a subReddit to follow the story:
https://www.reddit.com/r/Solarwinds/
I'm going to have to close out this posting without any pretence of
having reached grand conclusions: Possibly, more will come out.
However, if I had to guess, based on available evidence, the root cause
will turn out to involve SolarWinds, Inc. security incompetence --
made worse by the shortage of transparency that is typical with
proprietary software companies.
You might wonder: Could something similar happen with, for example, the
Debian Project? The simple answer is 'yes', but there is competent
management of key-signing both at the ftp-master build machines and
among the individual maintainers of Debian packages. Basically,
the all-volunteer Debian Project routinely does _way_ better than this
major-name software company did.
Alan Davis
Dec 17, 2020, 7:11:21 PM12/17/20
to berke...@googlegroups.com
My naive perspective runs something like this: the human brain evolved to solve problems, not so much for engineering air-tight systems, despite our self-portrayal as a species apart. Consequently, any system created by humans will suffer from our inability to perceive all possible aspects and vulnerabilities: it can never be guaranteed to be secure / airtight / escape-proof / unsinkable. In my view, that is a given. Scientific knowledge proceeds by increments, not generally by sudden paradigm-breaking developments. To wit, numerous simultaneous inventions: calculus, by Leibnitz and Newton; color photography in Europe and America---just two examples.
With regard to cyber-security, vis a vis breaches such as this---beside issues about proprietary software being inherently untrustworthy because it cannot be brainstormed, but seeks profit and/or security through obfuscation---any solution we propose is automatically exposed to the extraordinary ability of the human mind to solve problems. This seems to be evolutionary arms-race. We may notice calls from in-kind retribution. One cannot imagine a way out of this, Diligence, perhaps, and bearing in mind the inherent impracticality of technological solutions and protections. I am reminded of the advice that a computer system is only secure when it is locked with a meat-space lock and key. We should not be surprised when breaches occur. Finger pointing is futile.
This is just my personal point of view. I appreciate the somewhat comprehensive response by Rick Moen.
This morning I stumbled upon a podcast on Weekday Radio about a serious nuclear accident at Humboldt power station. The HBO series _Chernobyl_: is it pertinent to the (in)actions of SolarWinds? To the social and political undercurrents that lead us to rely on these? Remodeling the national security infrastructure alongside the commercialization of software and the Internet?
I am way over my head here, and probably contributing nothing.
Alan Davis
--
You received this message because you are subscribed to the Google Groups "BerkeleyLUG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylug...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/berkeleylug/20201217191540.GI28791%40linuxmafia.com.
--
"This ignorance about the limits of the earth's ability to absorb
pollutants should be reason enough for caution in the release
of polluting substances."
---Meadows et al. 1972. Limits to Growth. (p. 81) pollutants should be reason enough for caution in the release
of polluting substances."
Rick Moen
Dec 17, 2020, 8:21:44 PM12/17/20
to BerkeleyLUG
Some comments about links Aaron posted, composing these in-passim as I
read the articles in question:
Quoting goossbears (acoh...@gmail.com):
> - Gizmodo's 'Feds Still Trying to Determine How Screwed They Are After
> Massive SolarWinds Hack',
> https://gizmodo.com/feds-still-trying-to-determine-how-screwed-they-are-aft-1845888076
users of the Orion Platform software who got shot in the foot, and
partly because of the sensitive nature of what Orion Platform does,
i.e., network monitoring, which I gather requires that the software
run with high privilege and access to networks and remote systems.
As with other articles I've seen, here, Gizmodo reporter Tom McKay
doesn't appear to properly grapple with just how grossly negligent
SolarWinds, Inc. was revealed to have been. Anyone old enough will
remember how much flak Ford Motors took over supposedly exploding gas
tanks on Pintos (which turned out to be a greatly exaggerated story, but
that's not the point). Somehow, a major software company publishes
software that, when intalled at client sites, destroys client companys'
IT security, and all everyone can talk about is how freaked out the
customers are, i.e., hardly anyone's pointing an appropriate degree
of blame at the negligent party.
This astonishes me. It's like the botulism scandal over cans of Bon Vivant
vichyssoise, except with all the coverage being over the suffering of
victims and nobody saying anything about, looking at, or thinking about
the manufacturer -- and their attitude was just 'Well, sure, some cans
of soup poison and kill people, but, hey, stuff happens.'
Those responsible built a backdoor into Orion, an IT management
software produced by SolarWinds, possibly by breaking into Microsoft
email accounts and other systems, according to the Wall Street Journal.
[link] They then used it to contaminate software updates provided by the
company with malware in March and June 2020. In addition to U.S.
government agencies, the attackers also hit security firm FireEye;
senior vice president and chief technical officer, Charles Carmakal,
told Bloomberg [link] the firm was subsequently able to trace the intrusion
back to SolarWinds before it notified authorities.
The WSJ surmise was as follows: "How the hackers gained access to
SolarWinds systems to introduce the malicious code is still uncertain.
The company said that its Microsoft email accounts had been compromised
and that this access may have been used to glean more data from the
company’s Office productivity tools."
That's nothing like a complete picture on the vital "How did compromise
and privilege escalation occur?" question (not to mention obviously
involving speculation), but suggests a fatal laxness at SolarWinds, Inc.
Companies (and projects, like Debian) that take code-signing seriously
treat custody of the signing keys (and the production code repo) like
the crown jewels. There should have been _no_ path to get to them via
things like phishing and other dumb probes against Microsoft Exchange /
Microsoft Office.
> - CNN's Politics' article 'Massive hack of US government launches search
> for answers as Russia named top suspect',
> https://www.cnn.com/2020/12/16/politics/us-government-agencies-hack-uncertainty/index.html
This is what you get when you have an IT story covered by competent
political reports instead of competent IT reporters: You get an article
about _who_, when the question mainly of interest is _how_.
> - CNN's Business article 'Why the US government hack is literally keeping
> security experts awake at night',
> https://www.cnn.com/2020/12/16/tech/solarwinds-orion-hack-explained/index.html
Despite URL, this generally meritorious article likewise didn't explain
the Orion hack, except in the sense of saying "It was done via
SolarWinds's Orion Platform software, where the bad guys had full
control for over six months (BBC, below, says eight months) and
piggybacked their code to infiltrate customers via SolarWinds's signed
code for its retail software products."
> - The BBC News' Tech article 'SolarWinds Orion: More US government
> agencies hacked', https://www.bbc.com/news/technology-55318815
Reasonable layman's overview, adds some deserved praise for the response
of cybersecurity firm FireEyes.
> - The BBC News' Tech article 'SolarWinds: Why the Sunburst hack is so
> serious', https://www.bbc.com/news/technology-55321643
This is a piece by a different BBC reporter who makes quite a lot of
basic errors, which I probably shouldn't waste time listing. I have
some sympathy for IT/'tech' reporters, always expected to provide good
coverage on impossible deadlines. This article is mostly about
ramifications, and shows that the reporter called up a bunch of contacts
in relevant fields, and relied heavily on what they said. (That is not
a bad thing. I'm just saying it's that type of article.)
Bruce Schneier is reporting on the reporting.
https://www.schneier.com/blog/archives/2020/12/another-massive-russian-hack-of-us-government-networks.html
https://www.schneier.com/blog/archives/2020/12/how-the-solarwinds-hackers-bypassed-duo-multi-factor-authentication.html
https://www.schneier.com/blog/archives/2020/12/more-on-the-solarwinds-breach.html
Schneier makes the point -- obvious to me, but maybe not to most readers
-- that just replacing the trojaned software installed via SolarWinds's
oopsie with a non-trojaned version is not _nearly_ good enough: that
any/all of SolarWinds's ten of thousands of affected customers are going
to have to do _major_ work to rule out and correct persistent, ongoing
penetration of their networks and systems. Just removing or upgrading
the trojaned Orion Platform software is closing the barn door after the
horse escaped.
SolarWinds, Inc. retroactively deleted the public list of its customers
from its public-facing Web site shortly after the scandal hit, but,
well, too late, fellahs!
https://web.archive.org/web/20201214143046/https://www.solarwinds.com/company/customers
(As noted in the comments on Schneier's blog, Internet Archive is not a
foolproof repository of things that moneyed interests want to make go
away, in that they sometimes take down their mirror copies in response
to pressure.)
Another commenter says that the customer list captured by Internet
Archive / Wayback Machine is a "very small subset of the SolarWinds
clients".
Found via the second of the above-cited Schneier links:
https://arstechnica.com/information-technology/2020/12/solarwinds-hackers-have-a-clever-way-to-bypass-multi-factor-authentication/
ADVANCED PERSISTENT THREAT —
SolarWinds hackers have a clever way to bypass multi-factor
authentication
Hackers who hit SolarWinds compromised a think tank three separate
times.
DAN GOODIN - 12/14/2020
Oh? _That's_ interesting.
Article cites researchers at a security firm named Volexity who say
they'd encountered late last year / eartly this year the same attackers
who compromised SolarWinds, and noticed that they'd used a clever trick
to neuter multi-factor authentication on the attacked network of a
think-tank organisation. However, this hack required that the intruders
first possess 'Administrator' access on the target MS-Windows network,
i.e., root privilege. The target company used a two-factor
authentication system published by Duo Security. Having Administrator
access, they simply stole a Duo Security token file from the target
company's Outlook Web Access server, used that to generate a special
'cookie' file, and then figuratively waved that around like Doctor
Who's psychic paper to fake out authentication servers in a 'Oh, you
don't need the second factor in addition to my stolen username and
password' sense.
So, this isn't actually very surprising -- but it does underline the
point that, if subject to a comprehensive breach, the recovering
organisation must asssume that _all_ of the existing security
infrastructure is untrustworthy, including everyone's passwords and
system-internal security tokens.
It should be noted that both Volexity and FireEye are being careful
about the attacker's identity -- in distinction to many in the press who
are saying APT29 / Cosy Bear. Volexity merely calls the attacker Dark
Halo, and FireEye calls them UNC2452 - both names invented for the
purpose.
https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html
https://www.volexity.com/blog/2020/12/14/dark-halo-leverages-solarwinds-compromise-to-breach-organizations/
read the articles in question:
Quoting goossbears (acoh...@gmail.com):
> - Gizmodo's 'Feds Still Trying to Determine How Screwed They Are After
> Massive SolarWinds Hack',
> https://gizmodo.com/feds-still-trying-to-determine-how-screwed-they-are-aft-1845888076
A cyberattack that began by targeting an IT firm used by numerous
federal government agencies, Fortune 500 companies, and other high-value
targets is shaping up to be a historic event.
It appears so, partly because of the wide scope of SolarWinds, Inc.
federal government agencies, Fortune 500 companies, and other high-value
targets is shaping up to be a historic event.
users of the Orion Platform software who got shot in the foot, and
partly because of the sensitive nature of what Orion Platform does,
i.e., network monitoring, which I gather requires that the software
run with high privilege and access to networks and remote systems.
As with other articles I've seen, here, Gizmodo reporter Tom McKay
doesn't appear to properly grapple with just how grossly negligent
SolarWinds, Inc. was revealed to have been. Anyone old enough will
remember how much flak Ford Motors took over supposedly exploding gas
tanks on Pintos (which turned out to be a greatly exaggerated story, but
that's not the point). Somehow, a major software company publishes
software that, when intalled at client sites, destroys client companys'
IT security, and all everyone can talk about is how freaked out the
customers are, i.e., hardly anyone's pointing an appropriate degree
of blame at the negligent party.
This astonishes me. It's like the botulism scandal over cans of Bon Vivant
vichyssoise, except with all the coverage being over the suffering of
victims and nobody saying anything about, looking at, or thinking about
the manufacturer -- and their attitude was just 'Well, sure, some cans
of soup poison and kill people, but, hey, stuff happens.'
Those responsible built a backdoor into Orion, an IT management
software produced by SolarWinds, possibly by breaking into Microsoft
[link] They then used it to contaminate software updates provided by the
company with malware in March and June 2020. In addition to U.S.
government agencies, the attackers also hit security firm FireEye;
senior vice president and chief technical officer, Charles Carmakal,
told Bloomberg [link] the firm was subsequently able to trace the intrusion
back to SolarWinds before it notified authorities.
The WSJ surmise was as follows: "How the hackers gained access to
SolarWinds systems to introduce the malicious code is still uncertain.
The company said that its Microsoft email accounts had been compromised
and that this access may have been used to glean more data from the
company’s Office productivity tools."
That's nothing like a complete picture on the vital "How did compromise
and privilege escalation occur?" question (not to mention obviously
involving speculation), but suggests a fatal laxness at SolarWinds, Inc.
Companies (and projects, like Debian) that take code-signing seriously
treat custody of the signing keys (and the production code repo) like
the crown jewels. There should have been _no_ path to get to them via
things like phishing and other dumb probes against Microsoft Exchange /
Microsoft Office.
> - CNN's Politics' article 'Massive hack of US government launches search
> for answers as Russia named top suspect',
> https://www.cnn.com/2020/12/16/politics/us-government-agencies-hack-uncertainty/index.html
political reports instead of competent IT reporters: You get an article
about _who_, when the question mainly of interest is _how_.
> - CNN's Business article 'Why the US government hack is literally keeping
> security experts awake at night',
> https://www.cnn.com/2020/12/16/tech/solarwinds-orion-hack-explained/index.html
the Orion hack, except in the sense of saying "It was done via
SolarWinds's Orion Platform software, where the bad guys had full
control for over six months (BBC, below, says eight months) and
piggybacked their code to infiltrate customers via SolarWinds's signed
code for its retail software products."
> - The BBC News' Tech article 'SolarWinds Orion: More US government
> agencies hacked', https://www.bbc.com/news/technology-55318815
of cybersecurity firm FireEyes.
> - The BBC News' Tech article 'SolarWinds: Why the Sunburst hack is so
> serious', https://www.bbc.com/news/technology-55321643
basic errors, which I probably shouldn't waste time listing. I have
some sympathy for IT/'tech' reporters, always expected to provide good
coverage on impossible deadlines. This article is mostly about
ramifications, and shows that the reporter called up a bunch of contacts
in relevant fields, and relied heavily on what they said. (That is not
a bad thing. I'm just saying it's that type of article.)
Bruce Schneier is reporting on the reporting.
https://www.schneier.com/blog/archives/2020/12/another-massive-russian-hack-of-us-government-networks.html
https://www.schneier.com/blog/archives/2020/12/how-the-solarwinds-hackers-bypassed-duo-multi-factor-authentication.html
https://www.schneier.com/blog/archives/2020/12/more-on-the-solarwinds-breach.html
Schneier makes the point -- obvious to me, but maybe not to most readers
-- that just replacing the trojaned software installed via SolarWinds's
oopsie with a non-trojaned version is not _nearly_ good enough: that
any/all of SolarWinds's ten of thousands of affected customers are going
to have to do _major_ work to rule out and correct persistent, ongoing
penetration of their networks and systems. Just removing or upgrading
the trojaned Orion Platform software is closing the barn door after the
horse escaped.
SolarWinds, Inc. retroactively deleted the public list of its customers
from its public-facing Web site shortly after the scandal hit, but,
well, too late, fellahs!
https://web.archive.org/web/20201214143046/https://www.solarwinds.com/company/customers
(As noted in the comments on Schneier's blog, Internet Archive is not a
foolproof repository of things that moneyed interests want to make go
away, in that they sometimes take down their mirror copies in response
to pressure.)
Another commenter says that the customer list captured by Internet
Archive / Wayback Machine is a "very small subset of the SolarWinds
clients".
Found via the second of the above-cited Schneier links:
https://arstechnica.com/information-technology/2020/12/solarwinds-hackers-have-a-clever-way-to-bypass-multi-factor-authentication/
ADVANCED PERSISTENT THREAT —
SolarWinds hackers have a clever way to bypass multi-factor
authentication
Hackers who hit SolarWinds compromised a think tank three separate
times.
DAN GOODIN - 12/14/2020
Oh? _That's_ interesting.
Article cites researchers at a security firm named Volexity who say
they'd encountered late last year / eartly this year the same attackers
who compromised SolarWinds, and noticed that they'd used a clever trick
to neuter multi-factor authentication on the attacked network of a
think-tank organisation. However, this hack required that the intruders
first possess 'Administrator' access on the target MS-Windows network,
i.e., root privilege. The target company used a two-factor
authentication system published by Duo Security. Having Administrator
access, they simply stole a Duo Security token file from the target
company's Outlook Web Access server, used that to generate a special
'cookie' file, and then figuratively waved that around like Doctor
Who's psychic paper to fake out authentication servers in a 'Oh, you
don't need the second factor in addition to my stolen username and
password' sense.
So, this isn't actually very surprising -- but it does underline the
point that, if subject to a comprehensive breach, the recovering
organisation must asssume that _all_ of the existing security
infrastructure is untrustworthy, including everyone's passwords and
system-internal security tokens.
It should be noted that both Volexity and FireEye are being careful
about the attacker's identity -- in distinction to many in the press who
are saying APT29 / Cosy Bear. Volexity merely calls the attacker Dark
Halo, and FireEye calls them UNC2452 - both names invented for the
purpose.
https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html
https://www.volexity.com/blog/2020/12/14/dark-halo-leverages-solarwinds-compromise-to-breach-organizations/
Rick Moen
Dec 17, 2020, 9:35:16 PM12/17/20
to berke...@googlegroups.com
Quoting Alan Davis (alan3...@gmail.com):
> My naive perspective runs something like this:
> the human brain evolved to
> solve problems, not so much for engineering air-tight systems, despite our
> self-portrayal as a species apart. Consequently, any system created by
> humans will suffer from our inability to perceive all possible aspects and
> vulnerabilities....
> My naive perspective runs something like this:
> the human brain evolved to
> solve problems, not so much for engineering air-tight systems, despite our
> self-portrayal as a species apart. Consequently, any system created by
> humans will suffer from our inability to perceive all possible aspects and
Yeah, that's really not what you generally find. You find people /
organisations who mess up very fundamental things, and then major
collapses follow logically, and then the people / organisations point
fingers energetically in every _other_ direction.
> With regard to cyber-security, vis a vis breaches such as this---beside
> issues about proprietary software being inherently untrustworthy because it
> cannot be brainstormed, but seeks profit and/or security through
> obfuscation
Although most proprietary software happens to be binary-only for various
business and practical reasons, it's perfectly feasible to sell licences
to proprietary software with full source code visibility and build
instructions/tools granted to customers, or even to everyone.
For example, quite a bit of proprietary software implementations of
crypto are published with source code visibility, so outside experts can
audit it, making it more trustworthy to skeptical customers.
The central problem of proprietary code is actually _futureproofing_.
In a nutshell, it's about everyone, not just someone, having the right
to keep developing the code and using it for any purpose.
Let me give you an example: John Bradley's neat little tool 'xv' for
displaying and modifying graphics files on X11 Unixes was brilliant.
It probably still is. In early days of Linux (90s), there was nothing
better. It was small, fast, intuitive, robust, scripting-friendly (so
you could use it to, say, generate thumbnails from a directory of
thousands of files) -- and it came with full (small) C source code,
which anyone is permitted to distribute in unmodified form.
However: (1) John wants you to send him $35 if you use it and find it
useful, on the honour system. For personal use, this is merely
requested. For commercial/government/institutional use, it's required.
(2) Nobody else is granted the legal right to distribute modified
versions of xv.
John's most recent release was v. 3.10a, a quarter-century ago: It's
been abandonware since then. About two dozen people have sent John
fixes, which he's kind enough to make publicly available as source code
patches, but he isn't producing new versions. And, because he never
gave other people the legal right to distribute modified versions of xv,
nobody else can step in and release v. 3.11, no matter how many people
still love the program.
That's what I mean by no futureproofing. One guy had veto power over
whether the program ever got a new release, and all he had to do was
stop doing anything, and the program died. End of story.
> I am reminded of the advice that a computer system is only
> secure when it is locked with a meat-space lock and key.
security, and it leads to low standards and tolerance for massive
screw-ups -- worse, to not even bothering to figure out the substance of
what happened in a security incident. I politely disagree.
> We should not be surprised when breaches occur.
Your other vehicles?
Would you be surprised if all of those locks suddenly ceased to work and
everything you own got stolen?
> Finger pointing is futile.
Suppose the above happened, except it was just your Kryptonite brand
locks that suddenly ceased to work, all your other locks worked just
fine, and your family's bicycles and motorcycles got stolen. Would you
think it 'futile' to learn from experience and be dubious of that
company's competence?
> This morning I stumbled upon a podcast on Weekday Radio about a serious
> nuclear accident at Humboldt power station. The HBO series _Chernobyl_: is
> it pertinent to the (in)actions of SolarWinds?
http://betteridgeslaw.com/
Michael Paoli
Dec 20, 2020, 3:09:43 AM12/20/20
to BerkeleyLUG
> From: goossbears <acoh...@gmail.com>
> Subject: Security breach at multiple Federal agencies via SolarWinds
> Date: Thu, 17 Dec 2020 10:32:22 -0800 (PST)
> Further thoughts and insights on this from Michael P, Rick M, Thomas L, and
> anyone else here?
Okay, I'm not going to add a whole lot to this. Notably as it's already
been discussed fairly well elsewhere/elselist.
E.g. the conspire list.
http://linuxmafia.com/mailman/listinfo/conspire
http://linuxmafia.com/pipermail/conspire/2020-December/date.html
I think Rick Moen - and others, have already quite well covered it.
Though I might come from slightly different perspective,
and maybe if I were sufficiently motivated, I might find a few
minor bits to quibble slightly over, but I think Rick already pretty
much has it spot on well covered.
Reddit also has tons of stuff on SolarWinds too. "stuff" - great
expert highly qualified stuff, lots of the unwashed masses, ... it's
The Internet - there's quite the mix. I've mostly only barely skimmed
some of it, and certainly not tried to follow all of it, on Reddit
or elsewhere.
Lots of other places/sources too ... of, "of course", varying quality.
So, I'll add some bits and my commentary ... not necessarily in any
particular order - but maybe I'll try
First of all, yes, it's a big deal. When widely deployed
hardware/software/firmware/whatever is greatly used as critical part of
lots of IT infrastructure and quite widely deployed - rightly and/or
wrongly - and it's majorly compromised, it's a big deal.
This would tend to apply to stuff like - widely used/deployed
critical flaws in - operating systems, network equipment,
(")security(") software/hardware, management/monitoring/control software
and products, control systems, etc.
So, yep, suffice it to say this is quite bad.
How bad? Well, much of that depends not only how widely deployed,
and where, but how much - or little - one put trust into such.
Highly trust and put lots of faith and control into not-so-trustworthy
dubious stuff, and it can bite one - and very hard. Be much more
skeptical, don't trust anything too much, well use defense in depth
https://en.wikipedia.org/wiki/Defense_in_depth_(computing)
and then, well, not great, but at least not nearly so bad and the
damage is much more limited and controlled/isolated.
But hey, you know "all" (or at least so many of) those, e.g.
Federal agencies ... that typically get a "D" or "F" on their
report cards on security, and most of 'em don't get a "C" or
better ... well, what do you think happens with agencies like that
when they widely deploy something that ought be secure, but it's really
not that secure and reliable, and something really bad happens with it.
Yeah, ... that. Very bad.
Additionally, something gravely lacking here in so many places,
adequate monitoring and detection. This should'a been caught a whole
lot sooner and stopped and shut down ... but ... it wasn't.
Not that that's trivial to do, but that so many failed to do it,
yeah, that's also a big deal. That means a whole lot of nastiness
was going on a quite a long while before anybody noticed something
was up. So, that makes bad much worse.
One of the posts I saw mentioned, Debian, and "what if", as if what if
something like that had happened to Debian? Well, fair number of
years back, it kind'a did. A Debian Developer (DD)'s key was
compromised. And some bad folks started doing some nasty stuff - even
using a Zero-day
https://en.wikipedia.org/wiki/Zero-day_(computing)
exploit.
Well, unlike many others, Debian's got their sh*t together. They
detected this in highly short order (day or two or less?), shut that
sh*t down, clamped down tightly on everything, checked everything, fixed
all the issues and damage, and slowly and carefully reinstated all or
most all services that were in place before. So, since Debian caught it
so dang (relatively) fast, the damage was quite limited. There was
still a fair bit of clean-up, but much of that was mostly precautionary
over stuff that might've been exposed - what was actually altered was
relatively minimal and detected and shut down in quite short order.
Let's see ... I'm fuzzy on the details - it was years ago, so ... for
folks that may want to read more ...
https://www.debian.org/News/2003/20031202
... yeah, ... compromise to detection ... 29 hours.
And it was actually a sniffed password that was the initial vector into
the compromise.
SolarWinds ... what many months or more? Closer to 29 weeks than
29 hours. Yeah, seriously not good.
And yeah, everyone ought pay ample attention to security. But also,
many that are "jucier" targets ought pay a helluva lot more attention to
security, as they're much more attractive targets for attackers,
and the compromise risks are also much higher.
Yeah, ... once upon a time, I was working at a major financial
institution. SATAN
https://en.wikipedia.org/wiki/Security_Administrator_Tool_for_Analyzing_Networks
was released. And, within hours or less, said financial institution
saw being probed by SATAN. Well, said financial institution
pays attention - this did not go without notice. And follow-up.
It was tracked back. Somebody working somewhere else at some
other employer, without anyone's authorization and approval, and
way outside what they were supposed to be doing there, was using SATAN
to poke at and prod/scan said financial institution.
Well, said financial institution got in touch with the employer
from which the attacks were originating, it got tracked back to the
person doing it, and they were summarily instafired. (There may have
been further consequences/actions/outcomes, but I didn't specificly
hear).
Anyway, folks need pay attention, and especially jucier targets need pay
stringent attention and take appropriate measures - most notably to not
only try to prevent various attacks/compromises, and the like, but also
highly important - to detect such when they occur. Because there are
always threats, and not all will be fully prevented all the time ahead
of time. Stuff happens. E.g. insider job - how well does your security
software and hardware prevent that? Yeah, that's a tough one. Okay,
*when* it happens, can you at least detect that it happened? And
probably also with enough info to know who done it (or from whence it
came, and all possible available relevant details - like full traffic
captures of the stuff that happened?).
So ... a whole hella lot of SolarWinds users/customers, were not only
compromised by SolarWinds, but also, they generally failed to detect
that they'd been compromised and SolarWinds likewise failed to detect
the compromise, and unfortunately too for many of them,
lack of or insufficient defense in depth, the compromises went
relatively deep and insidious. So, yeah, a relatively big deal.
And, another thing. SolarWinds. People, companies, institutions,
whatever ... human(s) and/or run/operated by humans ... stuff happens,
thing break, folks make mistakes, screw up, whatever. Okay, not great,
but deal with it - fix/repair it as feasible, take the appropriate
actions as feasible to ensure it doesn't happen again, fess up to it,
tell the truth, promise to improve and actually do so, and move on.
But ... not SolarWinds. If there was any question that they were
sh*t before, it's no longer a question. What did they do?
They placed the blame for their own incompetency and screw ups
upon the innocent. Who/what did they blame and do they blame?
Did they blame themselves? No. Did they blame the attackers? No.
They blame Open Source. Well, f*ck SolarWinds, they've shown
themselves to be not only incompetent, but scum.
So, reminds me of recent news story. Major fuel pipeline leak.
And ... the news story blamed ... tree roots. Wrong! The tree roots
were doing what tree roots do, what they've evolved over millions
of years to do, and highly predictable. It's not like the
trees have a highly advanced well educated conscious society of
highly ethical trees, and, well, this one bad seed knew better
but intentionally misbehaved and screwed around with a fuel pipeline.
So, yeah, we're gonna send that bad tree to prison. Nope, that's
not how it works. The supposedly advanced civilization of humans
stupidly put a fuel pipeline where tree roots could get to and damage
and breach it, and not only did they do that, but they didn't
quickly detect and stop the problem, or even detect it before they
had a breach or major breach. So, the fault lies with those that
made poor decisions. Not with some tree roots.
Likewise SolarWinds. They got caught with their pants down.
I don't know if they failed to use suspenders or belts, or what
their problem is, but pants down ... and they're blaming gravity
for their gross exposure. Well, it ain't the fault of gravity.
It's not like the gravity of the situation couldn't be predicted.
And, yeah, doesn't help to have solarwinds123 as password on
external Internet accessible interfaces, either. SolarWinds
reeks of incompetence ... and clearly now also scum and sleaze,
if that weren't already clear before. And f*ck 'em for blaming
Open Source. I think they ought be forced to do any and all their
work henceforward with absolutely no use of open source in any
way whatsoever directly or indirectly. We can probably start
that by ripping out most or all of their network stacks,
most or all of their DNS access, probably most if not all of
the tools and software they use to build and test their
products, and sell and market them, etc., and keep going,
and rip out all the other Open Source out from under 'em,
and then lets see what they can do from there - let 'em
flounder and drown in nothing but their own rhetoric,
and limited to only using and interacting with
software that isn't Open Source and wasn't built,
developed, or delivered, using Open Source.
Oh yeah, and screw Equifax too - they likewise blamed Open Source
for their incompetence.
Well, was bit earlier, but in any case, SolarWinds blames
Open Source.
https://thwack.solarwinds.com/t5/Geek-Speak-Blogs/The-Pros-and-Cons-of-Open-source-Tools/ba-p/478665
"
Security becomes a major issue. Anyone can be hacked. However, the risk
is far less when it comes to proprietary software. Due to the nature of
open-source software allowing anyone to update the code, the risk of
downloading malicious code is much higher. One source referred to using
open-source software as "eating from a dirty fork." When you reach
in the drawer for a clean fork, you could be pulling out a dirty
utensil. That analogy is right on the money.
"
https://www.veracode.com/blog/security-news/are-we-eating-dirty-fork
"anyone to update the code". Oh really, tell me exactly how *anyone*
can change the code in the Debian Linux kernel I'm running.
Start 'splainin' to me, ... go ahead now.
And tell me again how your proprietary closed source code is so
much better because we can't look at it and we should trust you.
And likewise how your security practices aren't open for most or all
to inspect and that's so much better. How's that workin' out for you?
And your supply chain - don't have that open for inspection so we
can see if anything goes wrong. How's that going for you?
https://en.wikipedia.org/wiki/SolarWinds#2020_supply_chain_attack
"On December 13, 2020, The Washington Post reported"
"The company stated in an SEC filing that fewer than 18,000 of its
33,000 Orion customers were affected".
Oh, fewer that more than half of. Lovely. Odd double-speak way of
saying "most", or "the majority of".
"indications of compromise dating back to the spring of 2020"
$ awk '/^2020.*Vernal Equinox/ {print $1}' ~/calendar
2020-03-19
$ date -I -d '2020-03-19 + 29 weeks'
2020-10-08
$ awk '/^2020.*Summer Solstice / {print $1}' ~/calendar
2020-06-20
$ date -I -d '2020-06-20 + 29 weeks - 1 day'
2021-01-08
Yup, still looks to me a lot closer to 29 weeks than 29 hours.
Don't worry, that closed source stuff is only about 24x7=168 times or
so worse that Open Source.
"November 2019, a security researcher notified SolarWinds that their FTP
server had a weak password of 'solarwinds123', warning that 'any hacker
could upload malicious [files]' that would then be distributed to
SolarWinds customers."
Yep, quality operation there.
"SolarWinds" ... "employee passwords had been posted on GitHub in 2019."
Uh huh, ... top notch.
"SolarWinds said they would revoke the compromised certificates by
December 21, 2020".
Yep, ... right on top of it - major security breach, sure, we'll get
to fixin' the major bleeding in a bit over a week or so.
https://en.wikipedia.org/wiki/2020_United_States_federal_government_data_breach
"Trump" ... "on December 19, 2020, 'everything is well under control'"
Uh huh ... just like COVID-19. And it's gonna magically go away too,
right?
"had gone undetected for months" - yep, that's how you make bad lots
worse.
"allowed the attackers to" ... "perform federated authentication across
victim resources."
You put your trust in *what* software, and *what* level of trust?
And from what quality/integrity of software source? Oh, closed source
"trust me" model? What could go wrong?
"Within days of its discovery, at least 200 organizations around the
world had been found to be affected by the attack"
Don't worry, it's only most of Orion's 33,000 customers.
Nobody important like, ... oh, numerous Federal agencies, Microsoft,
FireEye, UK Home Office, the UK National Health Service,
the North Atlantic Treaty Organization (NATO), the European Parliament
... oops.
So, yeah, it's a big deal. It matters where one places one's trust, and
how much trust one places there. Good thing at least we've got a
president that we can ... oh sh*t.
> Subject: Security breach at multiple Federal agencies via SolarWinds
> Date: Thu, 17 Dec 2020 10:32:22 -0800 (PST)
> Further thoughts and insights on this from Michael P, Rick M, Thomas L, and
> anyone else here?
been discussed fairly well elsewhere/elselist.
E.g. the conspire list.
http://linuxmafia.com/mailman/listinfo/conspire
http://linuxmafia.com/pipermail/conspire/2020-December/date.html
I think Rick Moen - and others, have already quite well covered it.
Though I might come from slightly different perspective,
and maybe if I were sufficiently motivated, I might find a few
minor bits to quibble slightly over, but I think Rick already pretty
much has it spot on well covered.
Reddit also has tons of stuff on SolarWinds too. "stuff" - great
expert highly qualified stuff, lots of the unwashed masses, ... it's
The Internet - there's quite the mix. I've mostly only barely skimmed
some of it, and certainly not tried to follow all of it, on Reddit
or elsewhere.
Lots of other places/sources too ... of, "of course", varying quality.
So, I'll add some bits and my commentary ... not necessarily in any
particular order - but maybe I'll try
First of all, yes, it's a big deal. When widely deployed
hardware/software/firmware/whatever is greatly used as critical part of
lots of IT infrastructure and quite widely deployed - rightly and/or
wrongly - and it's majorly compromised, it's a big deal.
This would tend to apply to stuff like - widely used/deployed
critical flaws in - operating systems, network equipment,
(")security(") software/hardware, management/monitoring/control software
and products, control systems, etc.
So, yep, suffice it to say this is quite bad.
How bad? Well, much of that depends not only how widely deployed,
and where, but how much - or little - one put trust into such.
Highly trust and put lots of faith and control into not-so-trustworthy
dubious stuff, and it can bite one - and very hard. Be much more
skeptical, don't trust anything too much, well use defense in depth
https://en.wikipedia.org/wiki/Defense_in_depth_(computing)
and then, well, not great, but at least not nearly so bad and the
damage is much more limited and controlled/isolated.
But hey, you know "all" (or at least so many of) those, e.g.
Federal agencies ... that typically get a "D" or "F" on their
report cards on security, and most of 'em don't get a "C" or
better ... well, what do you think happens with agencies like that
when they widely deploy something that ought be secure, but it's really
not that secure and reliable, and something really bad happens with it.
Yeah, ... that. Very bad.
Additionally, something gravely lacking here in so many places,
adequate monitoring and detection. This should'a been caught a whole
lot sooner and stopped and shut down ... but ... it wasn't.
Not that that's trivial to do, but that so many failed to do it,
yeah, that's also a big deal. That means a whole lot of nastiness
was going on a quite a long while before anybody noticed something
was up. So, that makes bad much worse.
One of the posts I saw mentioned, Debian, and "what if", as if what if
something like that had happened to Debian? Well, fair number of
years back, it kind'a did. A Debian Developer (DD)'s key was
compromised. And some bad folks started doing some nasty stuff - even
using a Zero-day
https://en.wikipedia.org/wiki/Zero-day_(computing)
exploit.
Well, unlike many others, Debian's got their sh*t together. They
detected this in highly short order (day or two or less?), shut that
sh*t down, clamped down tightly on everything, checked everything, fixed
all the issues and damage, and slowly and carefully reinstated all or
most all services that were in place before. So, since Debian caught it
so dang (relatively) fast, the damage was quite limited. There was
still a fair bit of clean-up, but much of that was mostly precautionary
over stuff that might've been exposed - what was actually altered was
relatively minimal and detected and shut down in quite short order.
Let's see ... I'm fuzzy on the details - it was years ago, so ... for
folks that may want to read more ...
https://www.debian.org/News/2003/20031202
... yeah, ... compromise to detection ... 29 hours.
And it was actually a sniffed password that was the initial vector into
the compromise.
SolarWinds ... what many months or more? Closer to 29 weeks than
29 hours. Yeah, seriously not good.
And yeah, everyone ought pay ample attention to security. But also,
many that are "jucier" targets ought pay a helluva lot more attention to
security, as they're much more attractive targets for attackers,
and the compromise risks are also much higher.
Yeah, ... once upon a time, I was working at a major financial
institution. SATAN
https://en.wikipedia.org/wiki/Security_Administrator_Tool_for_Analyzing_Networks
was released. And, within hours or less, said financial institution
saw being probed by SATAN. Well, said financial institution
pays attention - this did not go without notice. And follow-up.
It was tracked back. Somebody working somewhere else at some
other employer, without anyone's authorization and approval, and
way outside what they were supposed to be doing there, was using SATAN
to poke at and prod/scan said financial institution.
Well, said financial institution got in touch with the employer
from which the attacks were originating, it got tracked back to the
person doing it, and they were summarily instafired. (There may have
been further consequences/actions/outcomes, but I didn't specificly
hear).
Anyway, folks need pay attention, and especially jucier targets need pay
stringent attention and take appropriate measures - most notably to not
only try to prevent various attacks/compromises, and the like, but also
highly important - to detect such when they occur. Because there are
always threats, and not all will be fully prevented all the time ahead
of time. Stuff happens. E.g. insider job - how well does your security
software and hardware prevent that? Yeah, that's a tough one. Okay,
*when* it happens, can you at least detect that it happened? And
probably also with enough info to know who done it (or from whence it
came, and all possible available relevant details - like full traffic
captures of the stuff that happened?).
So ... a whole hella lot of SolarWinds users/customers, were not only
compromised by SolarWinds, but also, they generally failed to detect
that they'd been compromised and SolarWinds likewise failed to detect
the compromise, and unfortunately too for many of them,
lack of or insufficient defense in depth, the compromises went
relatively deep and insidious. So, yeah, a relatively big deal.
And, another thing. SolarWinds. People, companies, institutions,
whatever ... human(s) and/or run/operated by humans ... stuff happens,
thing break, folks make mistakes, screw up, whatever. Okay, not great,
but deal with it - fix/repair it as feasible, take the appropriate
actions as feasible to ensure it doesn't happen again, fess up to it,
tell the truth, promise to improve and actually do so, and move on.
But ... not SolarWinds. If there was any question that they were
sh*t before, it's no longer a question. What did they do?
They placed the blame for their own incompetency and screw ups
upon the innocent. Who/what did they blame and do they blame?
Did they blame themselves? No. Did they blame the attackers? No.
They blame Open Source. Well, f*ck SolarWinds, they've shown
themselves to be not only incompetent, but scum.
So, reminds me of recent news story. Major fuel pipeline leak.
And ... the news story blamed ... tree roots. Wrong! The tree roots
were doing what tree roots do, what they've evolved over millions
of years to do, and highly predictable. It's not like the
trees have a highly advanced well educated conscious society of
highly ethical trees, and, well, this one bad seed knew better
but intentionally misbehaved and screwed around with a fuel pipeline.
So, yeah, we're gonna send that bad tree to prison. Nope, that's
not how it works. The supposedly advanced civilization of humans
stupidly put a fuel pipeline where tree roots could get to and damage
and breach it, and not only did they do that, but they didn't
quickly detect and stop the problem, or even detect it before they
had a breach or major breach. So, the fault lies with those that
made poor decisions. Not with some tree roots.
Likewise SolarWinds. They got caught with their pants down.
I don't know if they failed to use suspenders or belts, or what
their problem is, but pants down ... and they're blaming gravity
for their gross exposure. Well, it ain't the fault of gravity.
It's not like the gravity of the situation couldn't be predicted.
And, yeah, doesn't help to have solarwinds123 as password on
external Internet accessible interfaces, either. SolarWinds
reeks of incompetence ... and clearly now also scum and sleaze,
if that weren't already clear before. And f*ck 'em for blaming
Open Source. I think they ought be forced to do any and all their
work henceforward with absolutely no use of open source in any
way whatsoever directly or indirectly. We can probably start
that by ripping out most or all of their network stacks,
most or all of their DNS access, probably most if not all of
the tools and software they use to build and test their
products, and sell and market them, etc., and keep going,
and rip out all the other Open Source out from under 'em,
and then lets see what they can do from there - let 'em
flounder and drown in nothing but their own rhetoric,
and limited to only using and interacting with
software that isn't Open Source and wasn't built,
developed, or delivered, using Open Source.
Oh yeah, and screw Equifax too - they likewise blamed Open Source
for their incompetence.
Well, was bit earlier, but in any case, SolarWinds blames
Open Source.
https://thwack.solarwinds.com/t5/Geek-Speak-Blogs/The-Pros-and-Cons-of-Open-source-Tools/ba-p/478665
"
Security becomes a major issue. Anyone can be hacked. However, the risk
is far less when it comes to proprietary software. Due to the nature of
open-source software allowing anyone to update the code, the risk of
downloading malicious code is much higher. One source referred to using
open-source software as "eating from a dirty fork." When you reach
in the drawer for a clean fork, you could be pulling out a dirty
utensil. That analogy is right on the money.
"
https://www.veracode.com/blog/security-news/are-we-eating-dirty-fork
"anyone to update the code". Oh really, tell me exactly how *anyone*
can change the code in the Debian Linux kernel I'm running.
Start 'splainin' to me, ... go ahead now.
And tell me again how your proprietary closed source code is so
much better because we can't look at it and we should trust you.
And likewise how your security practices aren't open for most or all
to inspect and that's so much better. How's that workin' out for you?
And your supply chain - don't have that open for inspection so we
can see if anything goes wrong. How's that going for you?
https://en.wikipedia.org/wiki/SolarWinds#2020_supply_chain_attack
"On December 13, 2020, The Washington Post reported"
"The company stated in an SEC filing that fewer than 18,000 of its
33,000 Orion customers were affected".
Oh, fewer that more than half of. Lovely. Odd double-speak way of
saying "most", or "the majority of".
"indications of compromise dating back to the spring of 2020"
$ awk '/^2020.*Vernal Equinox/ {print $1}' ~/calendar
2020-03-19
$ date -I -d '2020-03-19 + 29 weeks'
2020-10-08
$ awk '/^2020.*Summer Solstice / {print $1}' ~/calendar
2020-06-20
$ date -I -d '2020-06-20 + 29 weeks - 1 day'
2021-01-08
Yup, still looks to me a lot closer to 29 weeks than 29 hours.
Don't worry, that closed source stuff is only about 24x7=168 times or
so worse that Open Source.
"November 2019, a security researcher notified SolarWinds that their FTP
server had a weak password of 'solarwinds123', warning that 'any hacker
could upload malicious [files]' that would then be distributed to
SolarWinds customers."
Yep, quality operation there.
"SolarWinds" ... "employee passwords had been posted on GitHub in 2019."
Uh huh, ... top notch.
"SolarWinds said they would revoke the compromised certificates by
December 21, 2020".
Yep, ... right on top of it - major security breach, sure, we'll get
to fixin' the major bleeding in a bit over a week or so.
https://en.wikipedia.org/wiki/2020_United_States_federal_government_data_breach
"Trump" ... "on December 19, 2020, 'everything is well under control'"
Uh huh ... just like COVID-19. And it's gonna magically go away too,
right?
"had gone undetected for months" - yep, that's how you make bad lots
worse.
"allowed the attackers to" ... "perform federated authentication across
victim resources."
You put your trust in *what* software, and *what* level of trust?
And from what quality/integrity of software source? Oh, closed source
"trust me" model? What could go wrong?
"Within days of its discovery, at least 200 organizations around the
world had been found to be affected by the attack"
Don't worry, it's only most of Orion's 33,000 customers.
Nobody important like, ... oh, numerous Federal agencies, Microsoft,
FireEye, UK Home Office, the UK National Health Service,
the North Atlantic Treaty Organization (NATO), the European Parliament
... oops.
So, yeah, it's a big deal. It matters where one places one's trust, and
how much trust one places there. Good thing at least we've got a
president that we can ... oh sh*t.
Rick Moen
Dec 21, 2020, 5:30:09 PM12/21/20
to BerkeleyLUG
Quoting Michael Paoli (Michae...@cal.berkeley.edu):
> I think Rick Moen - and others, have already quite well covered it.
> Though I might come from slightly different perspective,
> and maybe if I were sufficiently motivated, I might find a few
> minor bits to quibble slightly over, but I think Rick already pretty
> much has it spot on well covered.
Since as mentioned I _literally_ was trying to assess the news coverage
> I think Rick Moen - and others, have already quite well covered it.
> Though I might come from slightly different perspective,
> and maybe if I were sufficiently motivated, I might find a few
> minor bits to quibble slightly over, but I think Rick already pretty
> much has it spot on well covered.
in real time as I read it, I'd not be the least bit surprised at my
having missed things. And also, there will doubtless be plot twists.
Anyway, all I promised was a quick take.
> First of all, yes, it's a big deal.
research institutions, etc. now have to worry about undetermined amounts
of security compromise, and that is a Big Problem. It's a problem
equally if the institution decides it might have been rooted and decides
to do a ground-up rearchitecting and rebuild, if the institution elects
to shrug off the problem and bet that it'll get away with it, or -- most
likely -- if the institution takes a few half-assed ineffective steps
involving buying some more security wooga-wooga.
Anyone who's worked in the field has seen management go for dumb
wooga-wooga 99 times out of 100 -- and never learning from experience.
Like, for example, corporate board rooms all over the world are having
briefings where the point discussed is "We relied on SolarWinds Orion
Platform for network management and ran versions said to have been
trojaned, and have excised that software and changed all passwords,
but really have no idea whether systems have been intruded upon by
criminals, what systems, and whether we've now locked them out."
Rationally, the next point of discussion _ought_ to be "Why is our
internal monitoring and detection is so bad, that we cannot answer that
question?", but experience says it won't be.
Instead, they'll just at most buy a consulting package to issue a report
and make some recommendations, so management can claim they followed
"best practices".
http://linuxmafia.com/~rick/lexicon.html#best-practices
Best Practices
Making sure your blunders are popular ones. Rationally, this term
_should_ mean "methods that meet professional standards of competence
and due care", but tends instead to be a managerial code phrase
meaning "If anything goes wrong, I want to escape being a specific
target of blame by pointing out that our hapless cock-up was the same
one countless others made, too."
> Additionally, something gravely lacking here in so many places,
> adequate monitoring and detection. This should'a been caught a whole
> lot sooner and stopped and shut down ... but ... it wasn't.
This is exactly where FireEye stands out for its leadership.
> One of the posts I saw mentioned, Debian, and "what if", as if what if
> something like that had happened to Debian? Well, fair number of
> years back, it kind'a did. A Debian Developer (DD)'s key was
> compromised. And some bad folks started doing some nasty stuff - even
> using a Zero-day
> https://en.wikipedia.org/wiki/Zero-day_(computing)
> exploit.
http://linuxmafia.com/~rick/constructive-paranoia.html
In particular, I pointed out how and why the Debian Project detected
within one day the compromise of four of their machines _despite_ it
having been a zero-day kernel exploit, took immediate effective action,
and wrote an authoritative after-action postmortem report. Also, I
pointed out that the Debian package archive was not compromised by the
intruder, and why.
As I mention in the article, Gentoo Project was likewise hit by the same
kernel security bug and that the compromise was detected within an
_hour_ by effective use of an IDS and a file-integrity checker.
These were two all-volunteer geek projects, and yet, when challenged by
an existential security threat, they responded correctly and effectively
-- succeeding immediately to a worse threat than SolarWinds faced, and
where SolarWinds allegedly had absolutely no idea they'd vfailed at
their one job for something like eight months (until customer FireEye
briefed them).
> Well, was bit earlier, but in any case, SolarWinds blames
> Open Source.
> https://thwack.solarwinds.com/t5/Geek-Speak-Blogs/The-Pros-and-Cons-of-Open-source-Tools/ba-p/478665
How does crow taste, Greg W. Stuart?
This _partcular_ Greg W. Stuart appears to be one of a stable of
rent-a-pundits SolarWinds publishes.
https://orangematter.solarwinds.com/brains/ If I have found the correct
one in hunting around, this one is an ex-USAF guy who fell in love with
VMware while in the service, and then decided that he's an IT expert.
Reply all
Reply to author
Forward
0 new messages