raw_sha2_crypt extremely slow

[boa...@gmail.com]

I'm running flask_security which uses passlib.

When I create a user and encrypt it's password it takes about 2 seconds.

Running a profiler revealed that _raw_sha2_crypt takes 1.89 seconds to run

self run is 0.52 seconds

and is calling 656006 times to hashlib digest

Am I doing something wrong? Isn't it too much?

Thanks

Profiler output:

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

        1    0.000    0.000    1.891    1.891 /Users/boazackerman/sentinel-cloud/.env/lib/python2.7/site-packages/flask_security/utils.py:143(encrypt_password)

        1    0.000    0.000    1.891    1.891 /Users/boazackerman/sentinel-cloud/.env/lib/python2.7/site-packages/passlib/context.py:2458(encrypt)

        1    0.000    0.000    1.891    1.891 /Users/boazackerman/sentinel-cloud/.env/lib/python2.7/site-packages/passlib/utils/handlers.py:554(encrypt)

        1    0.000    0.000    1.891    1.891 /Users/boazackerman/sentinel-cloud/.env/lib/python2.7/site-packages/passlib/utils/handlers.py:1463(_calc_checksum)

        1    0.000    0.000    1.891    1.891 /Users/boazackerman/sentinel-cloud/.env/lib/python2.7/site-packages/passlib/utils/handlers.py:1450(_calc_checksum_backend)

        1    0.000    0.000    1.891    1.891 /Users/boazackerman/sentinel-cloud/.env/lib/python2.7/site-packages/passlib/handlers/sha2_crypt.py:342(_calc_checksum_builtin)

        1    0.520    0.520    1.891    1.891 /Users/boazackerman/sentinel-cloud/.env/lib/python2.7/site-packages/passlib/handlers/sha2_crypt.py:56(_raw_sha2_crypt)

   656006    0.704    0.000    0.704    0.000 {_hashlib.openssl_sha512}

   656006    0.666    0.000    0.666    0.000 {method 'digest' of '_hashlib.HASH' objects}

[Eli Collins]

That is a bit much.   Ideally a normal user login should take around 0.3 - 0.4s per call, not 1.9.

One problem is that there's so much variation between systems that the default settings are either too large and slow, or too small and weak.   For each release I've been trying to average a few different systems I use. 

However, that sample is weighted towards some VPS services I use, and I think they upgraded their cpu performance more than I expected.  I've had a few other reports that the latest release of passlib may have set things a little too high out there (one such report is Issue 60).  I'm hoping to get some free time in December to work up a new passlib release, and plan to lower the default settings on the hashes (and try to formalize some of what I'm about to write below). 

...............

But that's in the vague future, to help you right now:

Lowering the default rounds used by sha512_crypt down to 130100 should hit the sweet spot for your system (based on the defaults, and the timing information you provided, 656000 rounds / 1.89s * 0.375s = about 130100 rounds). 

Doing this requires reconfiguring the passlib CryptContext instance that flask_security is using.  I'm not that familiar with flask & flask_security's architecture, but I perused the flask_security code, and if I read it right...

If you get a hold of the flask `app` object after flask_security has loaded, `app.extensions['security'].pwd_context` should (?) be a passlib CryptContext instance.  If that's the case, inserting a line during your application startup, and calling  `pwd_context.update(sha512_crypt__rounds=130100, sha512_crypt__max_rounds=130100)` should cause any new hashes to use the desired number of rounds, and rehash users as they log in to have a lower rounds value.

(Mind you, if this is just your test system, I'd re-profile on production, and scale the value accordingly.  Higher values are always better, as long the delay isn't too obnoxious).

- Eli

--
You received this message because you are subscribed to the Google Groups "passlib-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to passlib-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[boa...@gmail.com]

Thanks. It worked...

[Boaz Ackerman]

Another quick question, because I'm a bit confiused.

I'm reading that SHA-512 has 80 rounds.

How did we get to numbers in the 100,000s?

I'm guessing it is not hash rounds, and I'll be happy for a quick explanation on what rounds are we talking about

Thanks again

--
You received this message because you are subscribed to a topic in the Google Groups "passlib-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/passlib-users/ECkcbC2zihE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to passlib-user...@googlegroups.com.

[Eli Collins]

The cryptographic hash SHA-512 internally has 80 rounds of the internal scrambling/compression function that it uses.

"sha512_crypt" on the other hand is a different thing -- it's a password hash which uses the SHA-512 hash internally as it's main building block.  You can think of it roughly as performing repeated rounds of the SHA-512 cryptographic hash on your password + a salt (the reality of what it does is a bit more involved).  The 100,000s range values are the number of rounds that sha512_crypt is applying, which corresponds roughly linearly to the time taken (It's designed to be scaled like this, so the time-cost can be increased as systems get faster; without having to design a new algorithm entirely).

And yeah, the name confusion is annoying.  Wish the designers had called it something different; but there's a long tradition of naming password hashes "something-crypt" :)

Hope that explains things! 

- Eli

Eli Collins el...@astllc.org Software Design & Consulting Assurance Technologies www.astllc.org

[Boaz Ackerman]

Great explanation. Thanks!

Sorry to bother you with another question (I'm feeling my questions are trivial but I can't find any decsent documentation of other things explaining these)

When I run verify/identity - I needs to know the rounds number, 

so after I create passwords (a user signed up) with a specific number of rounds, I can't change it. If I do change it - old rounds user will not be able to log in 

(or I'm missing something fundamental in the way verify works - exactly what I can't find :))

[Boaz Ackerman]

For now I discovered that older users hashes "know" the right number of rounds (I'm seeing the same delay for existing users, and faster delay for new ones)

Couldn't quite follow the code to figure out how it is determined - is it in the salt?

If you can clarify this part, I would be very grateful

Thanks again

[Eli Collins]

Sorry I missed your followup email; happy to explain things a bit! 

All the modern hashes in passlib (sha512_crypt, bcrypt, pbkdf2_sha512, etc) encode a bunch of things into "hash" that's stored in the database. 

I put "hash" in quotes because it actually includes a lot of stuff - an identifier, the rounds and salt that was used, as well as the actual hash digest.  Thus when .verify() is called, it uses whatever settings are w/in the hash string, so it recreates the digest precisely.  Each of the hash formats is documented, so you can see what each part means -- such as sha256_crypt's format & algorithm section (sha512_crypt has the same format, just a different identifier).

The default rounds setting (e.g. set by the 'sha512_crypt__rounds=130100' part in my original email) is primarily only used when new hashes are generated (when .encrypt is called).

For existing users, it's going to keep using the old value, since that's whats stored in their hash strings. 

(This is all something I'm hoping to get a good writeup into the manual some time). 

- Eli

[Boaz Ackerman]

Awesome! Thanks a lot!

[d.her...@gmail.com]

Just came across this same issue using when hashing passwords with sha512_crypt in Flask-security. Modifying sha512_crypt__rounds and sha512_crypt__max_rounds did the trick. Thanks!