[ANN] Linode compromise possibly affecting Clojars

[Phil Hagelberg]

Today Linode announced that their database was attacked[1]. Clojars is
hosted on Linode, and while we have no evidence that the attackers used
their access to break into the VPS instance which hosts Clojars, we
can't rule out the possibility. Other VPS instances[2] have been broken
into.

Apparently the attack happened two weeks ago. In order to confirm that
there was no attack, we want to verify checksums of all the artifacts we
can based on copies that were fetched before the attack.

If you run a private proxying internal repository for your company, you
can help us verify checksums. I'll be posting a follow-up soon with some
code you can use to calculate and publish checksums so we can
investigate discrepancies.

At this point you should be aware that there is some risk in continuing
to pull artifacts from Clojars while the verification is in process. If
you have a private proxying repository, you may want to disable the
Clojars source to avoid pulling in any new possibly-compromised
artifacts and possibly clear out any artifacts that were fetched within
the past two weeks.

If you can help out with the verification process, please join the
#leiningen channel on freenode or reply to me personally.

thanks,
Phil

[1] - http://blog.linode.com/2013/04/16/security-incident-update/

[2] - http://seclists.org/nmap-dev/2013/q2/3
http://p.hagelb.org/clojars-compromise-ann.html

[Rostislav Svoboda]

Whata an [ANN] !!!
Please change the subject to [linode-compromise "1.0.0"]
:)

[Phil Hagelberg]

Phil Hagelberg writes:

> If you run a private proxying internal repository for your company, you
> can help us verify checksums. I'll be posting a follow-up soon with some
> code you can use to calculate and publish checksums so we can
> investigate discrepancies.

Update: Hugo Duncan pointed out that the Clojars search indices contain
checksums in them, so I've written a script to verify a repository
against a copy of the index I have from the 22nd of April.

https://github.com/technomancy/clojars-verify/blob/daa26b39341b6d9dae54269ca0ac127eb0bb90c8/src/clojars/verify.clj

However, this gets me a huge number of false negatives (~39%) for
unknown reasons, so we'll be falling back to the original plan of
verification from old copies.

We have a few sources of backups from before the compromise, but if
anyone else has a copy of the repository from before the first of April
it wouldn't hurt to have additional sources of verification. Please
contact me by email or on IRC in the #leiningen freenode channel if so.

thanks,
Phil

[Phil Hagelberg]

Update: thanks to an older backup from Ivan Kozik, we've been able to verify the
integrity of all but 45 jars. It's likely these were legitimate redeployments by
the maintainers, but I'm going to be reviewing the diffs by hand.

I've attached a list of jars which haven't been verified. If your dependency
tree (visible with `lein deps :tree` under Leiningen 2) does not include any of
these then you should be completely clear. If it does you're probably still
safe; I hope to have these cleared by the end of the day.

From what we know from Linode, the exposed data could not have resulted in a
compromise of the box without a reboot in order to reset passwords or use the
rescue image, and we've confirmed that a reboot did not occur. So checking the
jar diffs is probably not necessary, but we'd rather be extra-careful.

thanks,
Phil

[Andrew Wagner]

Just wanted to say, awesome job with this. I appreciate your diligence!

amazonica/amazonica/0.1.3/amazonica-0.1.3.jar
amazonica/amazonica/0.1.4/amazonica-0.1.4.jar
amazonica/amazonica/0.1.5/amazonica-0.1.5.jar
antler/caribou-admin/0.10.0/caribou-admin-0.10.0.jar
bwo/monads/0.1.0/monads-0.1.0.jar
cc/qbits/alia/1.0.0-beta7/alia-1.0.0-beta7.jar
chlorine/chlorine/1.5.2.1/chlorine-1.5.2.1.jar
chlorine/chlorine/1.5.2.2/chlorine-1.5.2.2.jar
chlorine/chlorine/1.5.3/chlorine-1.5.3.jar
clj-diffmatchpatch/clj-diffmatchpatch/0.0.9.1/clj-diffmatchpatch-0.0.9.1.jar
clj-diffmatchpatch/clj-diffmatchpatch/0.0.9.3/clj-diffmatchpatch-0.0.9.3.jar
clj-diffmatchpatch/clj-diffmatchpatch/0.0.9/clj-diffmatchpatch-0.0.9.jar
clojurewerkz/archimedes/1.0.0-alpha3/archimedes-1.0.0-alpha3.jar
com/akolov/xelery/0.3.0/xelery-0.3.0.jar
com/narkisr/carmine/1.6.0/carmine-1.6.0.jar
com/narkisr/gelfino-client/0.4.0/gelfino-client-0.4.0.jar
conveyor-coffeescript/conveyor-coffeescript/0.1.4/conveyor-coffeescript-0.1.4.jar
core-cl2/core-cl2/0.7.0/core-cl2-0.7.0.jar
core-cl2/core-cl2/0.7.1/core-cl2-0.7.1.jar
docopt/docopt/0.6.1/docopt-0.6.1.jar
factual/c4/0.0.11/c4-0.0.11.jar
homestake-server/homestake-server/0.1/homestake-server-0.1.jar
info/yasuhisay/clj-utils/0.1.1/clj-utils-0.1.1.jar
info/yasuhisay/liblinear/0.0.1/liblinear-0.0.1.jar
lamina/lamina/0.5.0-beta15/lamina-0.5.0-beta15.jar
lein-haml-sass/lein-haml-sass/0.2.5/lein-haml-sass-0.2.5.jar
lein-package/lein-package/0.1.2/lein-package-0.1.2.jar
lib-noir/lib-noir/0.5.0/lib-noir-0.5.0.jar
luminus/lein-template/0.5.3/lein-template-0.5.3.jar
luminus/lein-template/0.5.5/lein-template-0.5.5.jar
me/arrdem/imprecise/me.arrdem.imprecise/0.1.0/me.arrdem.imprecise-0.1.0.jar
net/clojure/monads/1.0.1/monads-1.0.1.jar
nsfw/lein-template/0.5.2/lein-template-0.5.2.jar
org/flatland/phonograph/0.1.4/phonograph-0.1.4.jar
org/trpr/integration-rabbitmq/1.2.2/integration-rabbitmq-1.2.2.jar
org/trpr/platform-core/1.2.2/platform-core-1.2.2.jar
org/trpr/platform-integration/1.2.2/platform-integration-1.2.2.jar
rojat/rojat-arrows/0.0.6/rojat-arrows-0.0.6.jar
rojat/rojat-math/0.0.2/rojat-math-0.0.2.jar
rojat/rojat-math/0.0.3/rojat-math-0.0.3.jar
rst-format-parser/rst-format-parser/0.0.1/rst-format-parser-0.0.1.jar
service-hub/service-hub/1.0.3/service-hub-1.0.3.jar
stereotype/stereotype/0.2.1/stereotype-0.2.1.jar
stereotype/stereotype/0.2.2/stereotype-0.2.2.jar
thebusby/clj-aws-ec2/0.2.2/clj-aws-ec2-0.2.2.jar

[Phil Hagelberg]

Update: I've manually reviewed a diff[1] of all changes to jars
published since the intrusion. I found nothing suspicious in the diff,
but I did see a couple instances of bytecode in it. Two of them were
just bytecode being removed, but in one of them the bytecode changed
when the new copy was redeployed.

So the current status is that we've verified everything except
rst-format-parser. This seems to be a fairly obscure jar with only 21
downloads listed. But I've contacted the maintainer to ask him to either
verify the checksum or redeploy a known-good jar. Unless you're one of
the few people using this jar, you should be safe[2].

Happy hacking,
Phil

[1] - http://p.hagelb.org/clojars-republished.diff.html

[2] - By "safe" here, I mean "as safe as you were before the intrusion".
You're probably still trusting unsigned jars. We're working on
making it easier to have good reason to trust your dependencies,
but it's slow going.

[Phil Hagelberg]

Andrew Wagner writes:

> Just wanted to say, awesome job with this. I appreciate your diligence!

Thanks! Luckily part of my job at Heroku is to keep an eye out for this
kind of thing, so that's why I'm able to spend more cycles on it when
issues do arise. But Alex Osborne, Ivan Kozik, and Nelson Morris also
helped a lot.

-Phil

[Phil Hagelberg]

For the sake of completeness I've included Alex Osborne's analysis of
the situation below. (Alex runs Clojars.)

-Phil

--------------------------------------------------------------------

The really annoying thing about security is it's impossible to
conclusively prove at any time anything is secure if you assume a
sufficiently sophisticated attacker. All we can do is best effort.

Check my logic here. There are four categories of files to verify.

"The obvious jar overwrite": When a file is overwritten such that it's
mtime or size changed Ivan's backups move the original version into a
./overwritten/ directory for that timestamp. This means that any files
that were obviosuly overwritten after April 1 are in this subset:

https://gist.github.com/ato/459ee535125e6a5b5489[1]

We could try to verify these with jar diffs.

"The sneaky jar overwrite": However if a file was overwritten such that
the mtime and size didn't change, then Ivan would never have fetched the
new version. We can verify these by testings against Ivan's current
sha1s. I have done this and found no discrepancies. The only files that
didn't match were the expected, those updated today after Ivan had
calculated his hashes:

./nsfw/lein-template/0.5.2/lein-template-0.5.2.jar
./antler/lein-caribou/2.1.2/lein-caribou-2.1.2.jar

"The obvious new jar": The attacker may not have overwritten an existing
jar but added a new one (new version of a popular library say). I don't
see how we can verify these (as being legit or not). This would have a
lower value to the attacker compared to overwriting a popular existing
version.

"The sneaky new jar": If a new file was created but backdated (with an
mtime in the past) Ivan will have downloaded it but the "-a" option to
rsync will mean his timestamps are also backdated. This might be
detectable by diffing Ivan's copies of the all-jars metadata file I guess.

------

Now rather than modifying the repository immediately the attacker could
have installed a mechanism for uploading a malicious file at a later date:

* change passwords, ssh keys, groups or email addresses in clojars db
* steal the SSL private key (for later use in MITM attacks)
* backdoor the VM

db: I don't have an offsite db backup recently predating the attack. The
closest thing I have is a backup taken with Linode's own backup
infrastructure on April 7 which is too late to conclude much from. I
compared the April 7 db with todays and there were a number of password
changes since then, including some of the people that contacted us for
support about it.

ssl: If the attacker can perform MITM attacks against them most users
will be screwed anyway as they will be accessing other repositories like
Maven Central over unprotected HTTP. However if we do discover any
evidence of the VM being comprised I'll request it be revoked.

vm backdoor: I've checked the obvious things (shadow entries,
authorized_keys files, running processes). But there's no way to verify
this with high confidence. We could do a fresh OS install I suppose, I
was intending to upgrade to Ubuntu 12.04 sometime this year anyway.

------

Finally if we take Linode's statements at face value I think it's very
unlikely Clojars was compromised. They state that the attackers gained
access only to the web server not the VM hosts. The evidence presented
by "ryan" (who claimed to be one of the attackers) also only referred to
the web server and database.

The most straightforward ways for the web server to compromise a
customer VM:

* There is a form to reset the root password of a target VM, this
requires rebooting the VM as it modifies the filesystem offline.
* You can use the LISH password (cleartext in db) to get to the VM's
serial console. This doesn't directly get you in as the getty still
prompts you to login.
* You can reboot the VM against a rescue disk image which you control
through the console.

The clojars VM hasn't been rebooted in the last few months.

If Linode is wrong and the VM host was rooted a (somewhat sophisticated)
online attack would be possible via memory modification.

[Dave Sann]

+1