[Python-Dev] Deprecate crypt module and revert PR 3854

[Christian Heimes]

Hi,

in PR 3854 [1] Serhiy added blowfish, extended DES and NT-Hash to
Python's crypt mdodule. I vetoed against addition of the APIs because
all these hashing algorithms are not state of the art. Their quality
ranges from old to horribly, horriblye broken beyond any repair.

Shortly after the PR has landed, I was made aware that glibc has
deprecated crypt(3) API [2] and favor of an external library called
libxcrypt [3] from OpenWall Linux. I have patched Python 3.7 [4] to
support libxcrypt.

In light of deprecation of crypt(3) glibc function and bad quality of
hashing algorithms, I'd like to raise the motion to revert 3854 and
deprecate the crypt module. The whole module should be rather moved into
3rd party library that wraps xcrypt.

Regards,
Christian

[1] https://github.com/python/cpython/pull/3854
[2]
https://www.phoronix.com/scan.php?page=news_item&px=Fedora-28-libxcrypt-Plans
[3] https://github.com/besser82/libxcrypt
[4] https://bugs.python.org/issue32635
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Guido van Rossum]

I'm all for nudging people in the direction of xcrypt. I assume we can't just switch the C-level crypt with xcrypt and leave the Python API unchanged?

However until a usable solution exist (either in the stdlib or as 3rd party) I don't think we should deprecate anything (deprecating things before the replacement is ready is stressful for everyone involved).

I'm also not sure I agree with removing support for old hashes. By all means put in the docs that they are unsafe. But if someone has a database full of old hashes it would be nice to be able to at least read/verify it, right?

Was a release already made with blowfish, extended DES and NT-Hash? (And what's so bad with blowfish? It's mentioned in the heading of the xcrypt project too.)

Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

--

--Guido van Rossum (python.org/~guido)

[Serhiy Storchaka]

02.02.18 18:18, Guido van Rossum пише:

> I'm all for nudging people in the direction of xcrypt. I assume we can't
> just switch the C-level crypt with xcrypt and leave the Python API
> unchanged?
>
> However until a usable solution exist (either in the stdlib or as 3rd
> party) I don't think we should deprecate anything (deprecating things
> before the replacement is ready is stressful for everyone involved).
>
> I'm also not sure I agree with removing support for old hashes. By all
> means put in the docs that they are unsafe. But if someone has a
> database full of old hashes it would be nice to be able to at least
> read/verify it, right?
>
> Was a release already made with blowfish, extended DES and NT-Hash? (And
> what's so bad with blowfish? It's mentioned in the heading of the xcrypt
> project too.)

To clarify, extended DES and NT-Hash were not added. They were removed
from my PR after Christians request. Only the Blowfish method was added,
and it is so strong as SHA-2 methods. It is the only method supported on
OpenBSD.

This PR is not a single enhancement made in the crypt module recently. I
also extended tests and added support for configuring SHA-2 methods.
There is an open PR (not merged before 3.7b1 unfortunately) for using
crypt_r() instead of crypt(): https://bugs.python.org/issue28503.

If deprecate the crypt module, should modules pwd, grp and spwd be
deprecated too? The crypt module is needed for checking password hashes
provided by spwd.

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Nathaniel Smith]

On Feb 2, 2018 7:24 AM, "Christian Heimes" <chri...@python.org> wrote:

Shortly after the PR has landed, I was made aware that glibc has
deprecated crypt(3) API [2] and favor of an external library called
libxcrypt [3] from OpenWall Linux. I have patched Python 3.7 [4] to
support libxcrypt.

In light of deprecation of crypt(3) glibc function and bad quality of
hashing algorithms, I'd like to raise the motion to revert 3854 and
deprecate the crypt module. The whole module should be rather moved into
3rd party library that wraps xcrypt.

If the crypt module can just as well use libxcrypt, and that's how the distros are going to build it, then what's the advantage of deprecating it? Won't it just continue to work indefinitely?

-n

[Antoine Pitrou]

On Fri, 2 Feb 2018 16:23:20 +0100
Christian Heimes <chri...@python.org> wrote:
> Hi,
>
> in PR 3854 [1] Serhiy added blowfish, extended DES and NT-Hash to
> Python's crypt mdodule. I vetoed against addition of the APIs because
> all these hashing algorithms are not state of the art. Their quality
> ranges from old to horribly, horriblye broken beyond any repair.
>
> Shortly after the PR has landed, I was made aware that glibc has
> deprecated crypt(3) API [2] and favor of an external library called
> libxcrypt [3] from OpenWall Linux. I have patched Python 3.7 [4] to
> support libxcrypt.
>
> In light of deprecation of crypt(3) glibc function and bad quality of
> hashing algorithms, I'd like to raise the motion to revert 3854 and
> deprecate the crypt module.

Those are two separate proposals.

On the topic of reverting PR #3854, I don't see the point. Is Blowfish
more fragile than the other algorithms? If not, it sounds ok to add it.

On the topic of deprecating the crypt module, that doesn't sound like a
good idea right now. People may need to generate crypt()-compatible
output for various reasons, such as being able to automate system
administration tasks.

Regards

Antoine.

[Dan Stromberg]

On Fri, Feb 2, 2018 at 12:31 PM, Antoine Pitrou <soli...@pitrou.net> wrote:
> On Fri, 2 Feb 2018 16:23:20 +0100
> Christian Heimes <chri...@python.org> wrote:
>> Hi,
>>
>> in PR 3854 [1] Serhiy added blowfish, extended DES and NT-Hash to
>> Python's crypt mdodule. I vetoed against addition of the APIs because
>> all these hashing algorithms are not state of the art. Their quality
>> ranges from old to horribly, horriblye broken beyond any repair.
>>
>> Shortly after the PR has landed, I was made aware that glibc has
>> deprecated crypt(3) API [2] and favor of an external library called
>> libxcrypt [3] from OpenWall Linux. I have patched Python 3.7 [4] to
>> support libxcrypt.
>>
>> In light of deprecation of crypt(3) glibc function and bad quality of
>> hashing algorithms, I'd like to raise the motion to revert 3854 and
>> deprecate the crypt module.
>
> Those are two separate proposals.
>
> On the topic of reverting PR #3854, I don't see the point. Is Blowfish
> more fragile than the other algorithms? If not, it sounds ok to add it.

I'm no cryptographer, but I believe Blowfish's author recommends using
Twofish instead now.

> On the topic of deprecating the crypt module, that doesn't sound like a
> good idea right now. People may need to generate crypt()-compatible
> output for various reasons, such as being able to automate system
> administration tasks.

Encryption algorithms continue to be needed even after the time at
which they should no longer be used in new code. But they probably
should be documented as deprecated and warned about at runtime - after
there are suitable alternatives.

[Christian Heimes]

On 2018-02-02 18:05, Serhiy Storchaka wrote:
> 02.02.18 18:18, Guido van Rossum пише:
>> I'm all for nudging people in the direction of xcrypt. I assume we
>> can't just switch the C-level crypt with xcrypt and leave the Python
>> API unchanged?
>>
>> However until a usable solution exist (either in the stdlib or as 3rd
>> party) I don't think we should deprecate anything (deprecating things
>> before the replacement is ready is stressful for everyone involved).
>>
>> I'm also not sure I agree with removing support for old hashes. By all
>> means put in the docs that they are unsafe. But if someone has a
>> database full of old hashes it would be nice to be able to at least
>> read/verify it, right?
>>
>> Was a release already made with blowfish, extended DES and NT-Hash?
>> (And what's so bad with blowfish? It's mentioned in the heading of the
>> xcrypt project too.)
>
> To clarify, extended DES and NT-Hash were not added. They were removed
> from my PR after Christians request. Only the Blowfish method was added,
> and it is so strong as SHA-2 methods. It is the only method supported on
> OpenBSD.

Ah, I misinterpreted the subject of the PR. The closed PR still mentions
extended DES and NT-Hash. I'm sorry and blame my travel fatigue. The
email was written at the airport after I had a conversion with somebody
about new Python 3.7 features.

> This PR is not a single enhancement made in the crypt module recently. I
> also extended tests and added support for configuring SHA-2 methods.
> There is an open PR (not merged before 3.7b1 unfortunately) for using
> crypt_r() instead of crypt(): https://bugs.python.org/issue28503.

In general I'm all for more tests and improvements of existing modules.
However in this case Python 3.7 is sending wrong signals. For example
additional of blowfish was prominently features on the largest German
newsletter for IT. Both blowfish and SSHA (salted sha) are legacy
password hashing algorithms. Glibc has moved them out of the main
library for a good reason. (*)

> If deprecate the crypt module, should modules pwd, grp and spwd be
> deprecated too? The crypt module is needed for checking password hashes
> provided by spwd.

The pwd and grp module are fine. The modules use proper libc APIs that
are internally backed by NSS (libc's Name Service Switch, not Mozilla's
Network Security Service). APIs such getpwnam are defined and
standardized since POSIX.1-2001. The pwd and grp automatically work with
any configured user and group provider, even LDAP, IdM or Active Directory.

Fun fact: Golang programs are usually statically compiled and don't even
use libc. However Go's os/user package requires CGO and libc because it
has to interface with libc and NSS to acquire user and group information.

The spwd module is a different story. It's a direct interface to
/etc/shadown using Linux-only APIs. The shadow DB API requires root
permission. I think it even circumvents system security policies and
identity provider.

tl;dr
pwd + grp == good, required
crypt + spwd == bad

Regards,
Christian

(*) Most Linux distros never had blowfish in libc anyway.

[Christian Heimes]

On 2018-02-02 21:31, Antoine Pitrou wrote:
> On Fri, 2 Feb 2018 16:23:20 +0100
> Christian Heimes <chri...@python.org> wrote:
>> Hi,
>>
>> in PR 3854 [1] Serhiy added blowfish, extended DES and NT-Hash to
>> Python's crypt mdodule. I vetoed against addition of the APIs because
>> all these hashing algorithms are not state of the art. Their quality
>> ranges from old to horribly, horriblye broken beyond any repair.
>>
>> Shortly after the PR has landed, I was made aware that glibc has
>> deprecated crypt(3) API [2] and favor of an external library called
>> libxcrypt [3] from OpenWall Linux. I have patched Python 3.7 [4] to
>> support libxcrypt.
>>
>> In light of deprecation of crypt(3) glibc function and bad quality of
>> hashing algorithms, I'd like to raise the motion to revert 3854 and
>> deprecate the crypt module.
>
> Those are two separate proposals.
>
> On the topic of reverting PR #3854, I don't see the point. Is Blowfish
> more fragile than the other algorithms? If not, it sounds ok to add it.

Blowfish password hashing algorithms (also known as bcrypt) hasn't been
state of the art of a long time. Here is an old blog post from 2012 that
explains some of the issues with Blowfish:

http://www.unlimitednovelty.com/2012/03/dont-use-bcrypt.html

> On the topic of deprecating the crypt module, that doesn't sound like a
> good idea right now. People may need to generate crypt()-compatible
> output for various reasons, such as being able to automate system
> administration tasks.

IMO legacy support is not a good argument to keep the crypt module. The
passlib [1] package provides an excellent selection of legacy and modern
password derivation and hashing algorithms. As an admin you cannot rely
on the crypt module. libc's crypt() function usually provides a subset.
For example most Linux libc do not provide blowfish based hashing.

Christian

[1] https://pythonhosted.org/passlib/

[Christian Heimes]

On 2018-02-02 21:21, Nathaniel Smith wrote:
> On Feb 2, 2018 7:24 AM, "Christian Heimes" <chri...@python.org

libxcrypt is now an external library that must be installed separately.
It bloats CPython's dependency tree and the size of Python container
images for a legacy feature.

For Python 4.0 I'd like to drop some of the old, dead batteries and
include include useful batteries.

Christian

[Christian Heimes]

On 2018-02-02 17:18, Guido van Rossum wrote:
> I'm all for nudging people in the direction of xcrypt. I assume we can't
> just switch the C-level crypt with xcrypt and leave the Python API
> unchanged?
>
> However until a usable solution exist (either in the stdlib or as 3rd
> party) I don't think we should deprecate anything (deprecating things
> before the replacement is ready is stressful for everyone involved).
>
> I'm also not sure I agree with removing support for old hashes. By all
> means put in the docs that they are unsafe. But if someone has a
> database full of old hashes it would be nice to be able to at least
> read/verify it, right?
>
> Was a release already made with blowfish, extended DES and NT-Hash? (And
> what's so bad with blowfish? It's mentioned in the heading of the xcrypt
> project too.)

I answered some of your questions in other replies and will answer the
remaining concerns on Monday.

You suggested a 3rd party module. I have cloned the crypt module with
Serhiy's improvements and turned it into a standalone module with a
ctypes interface, https://github.com/tiran/legacycrypt . I'll release
the package as soon as I find time to polish the documentation and give
Serhiy his will deserved credit for his work.

Christian

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Guido van Rossum]

You can deprecate the crypt module, update the docs to explain it's outdated and point to other 3rd party solutions. A few years from now we may be able to delete it. (With an intermediate step of issuing a non-silent deprecation warning.) Until then just leave it be. Possibly that's what your desired outcome is anyway?

On Sat, Feb 3, 2018 at 4:07 AM, Christian Heimes <chri...@python.org> wrote:

On 2018-02-02 17:18, Guido van Rossum wrote:
> I'm all for nudging people in the direction of xcrypt. I assume we can't
> just switch the C-level crypt with xcrypt and leave the Python API
> unchanged?
>
> However until a usable solution exist (either in the stdlib or as 3rd
> party) I don't think we should deprecate anything (deprecating things
> before the replacement is ready is stressful for everyone involved).
>
> I'm also not sure I agree with removing support for old hashes. By all
> means put in the docs that they are unsafe. But if someone has a
> database full of old hashes it would be nice to be able to at least
> read/verify it, right?
>
> Was a release already made with blowfish, extended DES and NT-Hash? (And
> what's so bad with blowfish? It's mentioned in the heading of the xcrypt
> project too.)

I answered some of your questions in other replies and will answer the
remaining concerns on Monday.

You suggested a 3rd party module. I have cloned the crypt module with
Serhiy's improvements and turned it into a standalone module with a
ctypes interface, https://github.com/tiran/legacycrypt . I'll release
the package as soon as I find time to polish the documentation and give
Serhiy his will deserved credit for his work.

Christian