[Django] #36179: hexed strings in common passwords database are not handled

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Type: Bug
Status: new | Component:
| contrib.auth
Version: 5.1 | Severity: Normal
Keywords: | Triage Stage:
CommonPasswordValidator | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Hello,

The common passwords database file
(https://github.com/django/django/blob/main/django/contrib/auth/common-
passwords.txt.gz) used by CommonPasswordValidator contains hexed strings
like `$hex[617364666a6b6c3a]` on line 1679. That decodes to `asdfjkl:`
which I believe is a common password that was intended to be included in
the database. Another example is `$hex[2623323333363a]` on line 8616 that
decodes to `&#2336:`. I see that
https://gist.github.com/roycewilliams/226886fd01572964e1431ac8afc999ce
contains the line `
50334:72aff1cfd90a90fd4174eb6dfdff5df7bbbe7e5b:$HEX[617364666a6b6c3a]` and
`echo -n 'asdfjkl:' | sha1sum` produces
`72aff1cfd90a90fd4174eb6dfdff5df7bbbe7e5b`. CommonPasswordValidator does
not handle those hexed strings which I believe is wrong.

I propose to update the database file to decode the hexed values and
remove those that obviously can't be entered by a user.
--
Ticket URL: <https://code.djangoproject.com/ticket/36179>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------

Reporter: Michel Le Bihan | Owner: (none)

Type: Bug | Status: new
Component: contrib.auth | Version: 5.1

Severity: Normal | Resolution:

Keywords: | Triage Stage:
CommonPasswordValidator | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Michel Le Bihan:

Old description:

> Hello,
>
> The common passwords database file
> (https://github.com/django/django/blob/main/django/contrib/auth/common-
> passwords.txt.gz) used by CommonPasswordValidator contains hexed strings
> like `$hex[617364666a6b6c3a]` on line 1679. That decodes to `asdfjkl:`
> which I believe is a common password that was intended to be included in
> the database. Another example is `$hex[2623323333363a]` on line 8616 that
> decodes to `&#2336:`. I see that
> https://gist.github.com/roycewilliams/226886fd01572964e1431ac8afc999ce
> contains the line `
> 50334:72aff1cfd90a90fd4174eb6dfdff5df7bbbe7e5b:$HEX[617364666a6b6c3a]`
> and `echo -n 'asdfjkl:' | sha1sum` produces
> `72aff1cfd90a90fd4174eb6dfdff5df7bbbe7e5b`. CommonPasswordValidator does
> not handle those hexed strings which I believe is wrong.
>
> I propose to update the database file to decode the hexed values and
> remove those that obviously can't be entered by a user.

New description:

Hello,

The common passwords database file
(https://github.com/django/django/blob/main/django/contrib/auth/common-
passwords.txt.gz) used by CommonPasswordValidator contains hexed strings
like `$hex[617364666a6b6c3a]` on line 1679. That decodes to `asdfjkl:`
which I believe is a common password that was intended to be included in
the database. Another example is `$hex[2623323333363a]` on line 8616 that
decodes to `&#2336:`. I see that
https://gist.github.com/roycewilliams/226886fd01572964e1431ac8afc999ce
contains the line
`50334:72aff1cfd90a90fd4174eb6dfdff5df7bbbe7e5b:$HEX[617364666a6b6c3a]`
and `echo -n 'asdfjkl:' | sha1sum` produces
`72aff1cfd90a90fd4174eb6dfdff5df7bbbe7e5b`. CommonPasswordValidator does
not handle those hexed strings which I believe is wrong.

I propose to update the database file to decode the hexed values and
remove those that obviously can't be entered by a user.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:1>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: (none)
Type: Bug | Status: new
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage:
CommonPasswordValidator | Unreviewed

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Michel Le Bihan):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:2>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------

Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: assigned

Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage:
CommonPasswordValidator | Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Antoliny):

* owner: (none) => Michel Le Bihan
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:3>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan

Type: Bug | Status: closed
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution: invalid

Keywords: | Triage Stage:
CommonPasswordValidator | Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* resolution: => invalid
* status: assigned => closed

Comment:

Thank you for the ticket
I think you are referring to there passwords
[https://gist.github.com/roycewilliams/226886fd01572964e1431ac8afc999ce
#file-pwnedpasswords-v6-top20k-txt-L7958 asdfjkl;] and
[https://gist.github.com/roycewilliams/226886fd01572964e1431ac8afc999ce
#file-pwnedpasswords-v6-top20k-txt-L8918 &#2336;] (`;` not `:`) which are
validating correctly, the others are not in the list
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:4>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: closed
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage:
CommonPasswordValidator | Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Michel Le Bihan):

Hello,

I'm referring to those with `:` at the end. Please just search `$hex` in
that file and you will notice the issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:5>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan

Type: Bug | Status: new

Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:

Keywords: | Triage Stage:
CommonPasswordValidator | Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Michel Le Bihan):

* resolution: invalid =>
* status: closed => new

--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:6>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: new
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage:
CommonPasswordValidator | Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Michel Le Bihan):

Please see this line:
https://gist.github.com/roycewilliams/226886fd01572964e1431ac8afc999ce
#file-pwnedpasswords-v6-top20k-txt-L1728
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:7>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: new
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
CommonPasswordValidator |

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* stage: Unreviewed => Accepted

Comment:

Ah thank you!
Folks can de-hex using this tool:
https://www.rapidtables.com/convert/number/hex-to-ascii.html

Here is a regression test:
{{{#!diff
--- a/tests/auth_tests/test_validators.py
+++ b/tests/auth_tests/test_validators.py
@@ -273,6 +273,15 @@ class CommonPasswordValidatorTest(SimpleTestCase):
CommonPasswordValidator().validate("godzilla")
self.assertEqual(cm.exception.messages, [expected_error])

+ def test_common_hexed_codes(self):
+ expected_error = "This password is too common."
+ common_hexed_passwords = ["asdfjkl:", "&#2336:"]
+ for password in common_hexed_passwords:
+ with self.subTest(password=password):
+ with self.assertRaises(ValidationError) as cm:
+ CommonPasswordValidator().validate(password)
+ self.assertEqual(cm.exception.messages, [expected_error])
+
def test_validate_custom_list(self):
path = os.path.join(
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:8>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: new
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
CommonPasswordValidator |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Michel Le Bihan):

I sent a patch that unhexes the values in the database:
https://github.com/django/django/pull/19155
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:9>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: new
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
CommonPasswordValidator |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Raphael Gaschignard):

* needs_better_patch: 0 => 1

Comment:

While most of the `$hex` entries are removed, there's still one entry
that's just `$hex`. I _believe_ this is an oversight, and not
intentional.
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:10>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: new
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
CommonPasswordValidator | checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Raphael Gaschignard):

* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin

Comment:

Per discusison in the PR, seems like this is actually part of the dataset.
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:11>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: new
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
CommonPasswordValidator |

Has patch: 1 | Needs documentation: 0

Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* needs_tests: 0 => 1
* stage: Ready for checkin => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:12>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: new
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
CommonPasswordValidator | checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):

* needs_tests: 1 => 0

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:13>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan

Type: Bug | Status: closed
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
CommonPasswordValidator | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed
* status: new => closed

Comment:

In [changeset:"727731d76d9dfd5304d536478d862778f6dd6d9b" 727731d]:
{{{#!CommitTicketReference repository=""
revision="727731d76d9dfd5304d536478d862778f6dd6d9b"
Fixed #36179 -- Unhexed entries and removed duplicates in auth/common-
passwords.txt.gz.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:14>

[Django]

#36179: hexed strings in common passwords database are not handled
-------------------------------------+-------------------------------------
Reporter: Michel Le Bihan | Owner: Michel Le
| Bihan
Type: Bug | Status: closed
Component: contrib.auth | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
CommonPasswordValidator | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"cb2ab4ee840e93168d6b4820c5443e1736296ab6" cb2ab4e]:
{{{#!CommitTicketReference repository=""
revision="cb2ab4ee840e93168d6b4820c5443e1736296ab6"
[5.2.x] Fixed #36179 -- Unhexed entries and removed duplicates in auth
/common-passwords.txt.gz.

Backport of 727731d76d9dfd5304d536478d862778f6dd6d9b from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36179#comment:15>