Security - attempted hack on Kenneth Reitz

[Thomas Kluyver]

Someone tried to get access to Kenneth Reitz's Github account, possibly with the intent of tampering with requests. As maintainers of another prominent Python project, we should be vigilant with our own accounts.

We already asked everyone with commit rights to enable two-factor authentication. This demonstrates that was a good idea. But we should be aware of other attack vectors, like DNS providers and PyPI (you typically store PyPI passwords in plain text in a hidden .pypirc file).

[Big Stone]

pypi , R , anaconda are showing mainly md5 on their website (and plumbing?) : shouldn't they use and popularize stronger hash technics ?

[Thomas Kluyver]

At least on PyPI, I believe that the hashes are more intended to check download integrity than security. The new PyPI that's in the works does appear to show SHA256 hashes (see https://pypi.org/project/ipython/#files ). But the strongest hash function in the world won't protect us if someone steals our PyPI login credentials and uses them to upload valid releases.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/acdc4d6d-373a-410d-b9e8-7f01e29b287b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kyle Kelley]

Thanks for the diligence here.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAOvn4qg6icTxNzJBjAm7BCLiGLnFHZKjvwaHMoq5ujj7rZ1FJQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Kyle Kelley (@rgbkrk; lambdaops.com)

[Matthias Bussonnier]

Hi all,

Thanks Thomas for starting the thread on the mailing list. And Big,
for pointing the hashes.

> We already asked everyone with commit rights to enable two-factor authentication

Is that true for all repositories ? I know that at least all
organisations owners have 2FA,
as owner have inherently push rights on all the repositories, for
individuals on teams and repos,
I'm not certain. Should we start a process enforce for everyone ? We
can also turn on Protected
branches, that force master to be modified only through PRs, which is
less strict, but can still block
some workflows.


I've been considering signing my git commits as well as the releases
with a PGP key.
I need to re-create one. I haven't investigated how to automatically
verify these though.
And if there are common practices on the Python world to deal with that.

I would also like at some point – in a hypothetic future – to
investigate reproducible build[1], to check
that different users/machines building the same commit would get the
same hashes, why not with
an option on warehouse/pypi to "+1 I have the same hash"/"-1 not the
same hash" to
have an indicator of whether there is something fishy.

Note also that last time I checked, git does not check the integrity
of repository while pushing/pulling by default.
Which mean that a commits could have been tampered with, without
changing its recorded hash.
It that the case it's easy to have that unnoticed [2].


Some people suggest to issue a

$ git config --global transfer.fsckobjects true

To make that permanent.

It might make cloning a bit slower, and **some** already existing
repositories, are actually corrupted,
so you won't be able to clone them before turning that option off.

Sidenote, I feel like these kind of questions are coming up more and
more regularly at SciPy,
in particular this year during the CondaForge BOF for example.

--
M

[1]: flit does reproducible builds.
[2]: I did experience non-matching hash with likely a bit flip, where
the actual commit hash was wrong, but git was not complaining until a
manual fsck.

On Wed, Aug 10, 2016 at 3:55 PM, Thomas Kluyver <tak...@gmail.com> wrote:

> At least on PyPI, I believe that the hashes are more intended to check
> download integrity than security. The new PyPI that's in the works does
> appear to show SHA256 hashes (see https://pypi.org/project/ipython/#files ).
> But the strongest hash function in the world won't protect us if someone
> steals our PyPI login credentials and uses them to upload valid releases.
>
> On 10 August 2016 at 23:01, Big Stone <stone...@gmail.com> wrote:
>>
>> pypi , R , anaconda are showing mainly md5 on their website (and
>> plumbing?) : shouldn't they use and popularize stronger hash technics ?
>>
>> On Wednesday, August 10, 2016 at 11:09:14 PM UTC+2, takowl wrote:
>>>
>>> Someone tried to get access to Kenneth Reitz's Github account, possibly
>>> with the intent of tampering with requests. As maintainers of another
>>> prominent Python project, we should be vigilant with our own accounts.
>>>
>>> We already asked everyone with commit rights to enable two-factor
>>> authentication. This demonstrates that was a good idea. But we should be
>>> aware of other attack vectors, like DNS providers and PyPI (you typically
>>> store PyPI passwords in plain text in a hidden .pypirc file).
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Project Jupyter" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to jupyter+u...@googlegroups.com.

>> To post to this group, send email to jup...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/acdc4d6d-373a-410d-b9e8-7f01e29b287b%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to jupyter+u...@googlegroups.com.

> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jupyter/CAOvn4qg6icTxNzJBjAm7BCLiGLnFHZKjvwaHMoq5ujj7rZ1FJQ%40mail.gmail.com.

[Fernando Perez]

Hi all,

I'd like to separate two things here: 

a) basic good practices with repo security by asking for 2FA for all committers, and 

b) changing our build/release process with high security in mind.

On Wed, Aug 10, 2016 at 4:40 PM, Matthias Bussonnier <bussonnie...@gmail.com> wrote:

Should we start a process enforce for everyone ? We can also turn on Protected branches, that force master to be modified only through PRs, which is less strict, but can still block some workflows.

This is about (a), and I would like to start by *very strongly encouraging* everyone who has commit rights to do this.  For reference, this is the incident Thomas referred to

http://www.kennethreitz.org/essays/on-cybersecurity-and-being-targeted

It is recommended that you use something other than SMS for two-factor but you can use that if that is the only viable option.  There are instructions here:  https://github.com/blog/1614-two-factor-authentication

You should also make sure that you retrieve and save the backup recovery codes, in case you find yourself needing to access your account without your mobile device on you, out of batteries, etc.  If anyone is curious, not using SMS is the official guidance provided by NIST, which is the agency that deals with crypto standards for the US government and public:

https://pages.nist.gov/800-63-3/sp800-63b.html

specifically, section 5.1.3.2. Out of Band Verifiers states:

Due to the risk that SMS messages may be intercepted or redirected, implementers of new systems SHOULD carefully consider alternative authenticators.  [...] . OOB using SMS is deprecated, and may no longer be allowed in future releases of this guidance.

 

We can then revisit this later and see if it's worth tightening further and requiring 2FA.

I've been considering signing my git commits as well as the releases
with a PGP key.  I need to re-create one. I haven't investigated how to automatically
verify these though.  And if there are common practices on the Python world to deal with that.

This is really (b).  Good security practices in the complete build/release/distribution chain is a very complex topic that none of the core devs are experts in.  Rather than trying things here and there, I think that if we're going to do something, it should be because someone volunteers to take the lead on really learning what the best accepted process is today, and then providing a clear set of guidelines for us to follow, vetted by others.  Until then, it feels like just turning knobs at random without a clear understanding of where the actual weak links in the chain are.

So on (b), I'd hold off for now on engaging any serious effort and keep on monitoring what other projects do.  Now, if anyone is going to volunteer to draft such a guide that would meaningfully improve things for us and that is solid/vetted enough to be accepted by other similar projects, great! But until then, I don't see the value in burning our scarce cycles here.

Cheers

f

--

Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

[Matthias Bussonnier]

Thanks Fernando,

I agree, that (a) and (b) are different things, I went into the (b)
description mostly in response to Big Stone who pointed out the
hashes, and wanted to clarify what could be done for that (which is
package signing). Sorry if it was off-topic.

I have opened https://github.com/jupyter/jupyter/issues/194 to remind
us to take care of (a).

Related resources for those interested in the subject of security and
python packaging:

A few years ago some projects were presented at SciPy and tried to
address some concern about tampering with packages, one of those was
peep[1], which functionality have been merged into pip8.

--
M




[1] https://pypi.python.org/pypi/peep/
[2] https://pip.readthedocs.io/en/stable/reference/pip_install/#hash-checking-mode

> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jupyter+u...@googlegroups.com.
> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jupyter/CAHAreOrNLT_6ahmm7SeKsVwBnF8vKcjD4GzChS8hJw8L7Gts7Q%40mail.gmail.com.

[Thomas Kluyver]

On 11 August 2016 at 01:39, Fernando Perez <fpere...@gmail.com> wrote:

a) basic good practices with repo security by asking for 2FA for all committers, and 

But I think there's an important point here that Github 2FA, while it's definitely a good security switch to flip, is not the only thing we need. We should think about all the pieces that might be vulnerable:

- Someone may try to gain access to our emails to reset passwords on other sites. This is what happened to Kenneth. Since most sites offer a password reset by email, their security can only be as strong as the security around our emails.

- Someone may try to steal SSH keys, which let you push to Github bypassing 2FA. We should remove SSH keys we're no longer using from our Github accounts, and if there's any reason to think private keys may have been compromised, we should regenerate them.

- Someone may try to steal PyPI credentials and upload malicious packages. PyPI does not offer 2FA (yet). If you upload packages from a machine which other people can access, check that your ~/.pypirc file is not world-readable. Try to log in to PyPI periodically so that you will notice if your password has been changed.

This is still separate from (b) - this is not about designing new security measures, it's about taking care that the existing ones are not undermined by our mistakes.

Thomas

[Fernando Perez]

On Thu, Aug 11, 2016 at 2:21 AM, Thomas Kluyver <tak...@gmail.com> wrote:

- Someone may try to gain access to our emails to reset passwords on other sites. This is what happened to Kenneth. Since most sites offer a password reset by email, their security can only be as strong as the security around our emails.

Correct, though fortunately 2FA did help in this case: the attacker did reset his password successfully, but they couldn't log in b/c he had 2FA on.

 

- Someone may try to steal SSH keys, which let you push to Github bypassing 2FA. We should remove SSH keys we're no longer using from our Github accounts, and if there's any reason to think private keys may have been compromised, we should regenerate them.

Yes, this does allow for code compromises.  2FA does protect against certain administrative actions that can only be accomplished via the UI.  But what you say about keys is certainly good practice we should follow.  I have a github-specific SSH key I don't reuse for other things.

 

- Someone may try to steal PyPI credentials and upload malicious packages. PyPI does not offer 2FA (yet). If you upload packages from a machine which other people can access, check that your ~/.pypirc file is not world-readable. Try to log in to PyPI periodically so that you will notice if your password has been changed.

Yup, in this sense PyPI is  one of the weakest links in the chain (no 2FA, passwords stored locally in plaintext, etc).  Good reminder.

[Jonathan Frederic]

I think a few months ago Jason made an effort to audit our permissions, that was probably a good idea. In light of this, I'm willing to give up pypi authorization to many of our projects, but I should probably stay on ipywidgets,just in case...  I never liked publishing to pypi anyways ;)

--

You received this message because you are subscribed to the Google Groups "Project Jupyter" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAHAreOo0nBBx9hYMfY76tctzgunB3xttYeiuZVJN6othY7X1zA%40mail.gmail.com.

[Thomas Kluyver]

On 12 August 2016 at 01:59, Jonathan Frederic <jon.f...@gmail.com> wrote:

I think a few months ago Jason made an effort to audit our permissions, that was probably a good idea. In light of this, I'm willing to give up pypi authorization to many of our projects, but I should probably stay on ipywidgets,just in case...  I never liked publishing to pypi anyways ;)

Thanks Jon; I've just removed you from notebook, qtconsole, nbconvert and nbformat, which were the projects I could find you on. This needn't be permanent - we can always add you back if you agree to take a turn as release manager for any project.