Number of tries and delay with external authentication with multiple authenticators?

[Andreas Ntaflos]

Hi Jan (this list seems to have only one -- now two -- member at the
moment :)),

thank you very much for implementing my suggestion (multiple
AuthExternal authenticators). I finally had the opportunity to test it
and it seems to work just fine! I have a question though: there seems
to be a limit how often one authenticator is tried until it is
considered to have failed and the next is used. This limit is set to
20, is that correct? Is it configurable somewhere?

I ask because when I have

AuthBasicProvider external
AuthExternal pwauth mysql-auth

in my Apache configuration each try of the pwauth authenticator seems
to take about 1-2 seconds to complete (and fail) if the user in
question is only available through the second authenticator (mysql-
auth in this case). This results in a delay of about 20-40 seconds
until the next authenticator is asked. A regular user, unaware of the
details, would probably think that a network problem or some other
unexpected error occurred and would likely cancel the whole
authentication request.

Reversing the order and making mysql-auth the first authenticator
mitigates that problem since 20 database queries usually don't take
even a second to complete. The delay is thus not noticeable. But is
there anything to be done except putting pwauth last? Or did I miss
something else?

Thanks again!

Andreas

[Jan Wolter]

On Wed, 2009-06-10 at 15:53 -0700, Andreas Ntaflos wrote:
> Hi Jan (this list seems to have only one -- now two -- member at the
> moment :)),

Yup. The list is fairly new. I'm trying to evolve mod_auth_external
into a state where it isn't entirely dependent on me to maintain it, and
provide support for it, so I've shifted to hosting the project on
code.google.com and added the mailing list. Not much has come of this
yet, but that's OK. I have no plans to stop supporting it myself.

Glad you joined though. Being the only person on a mailing list is kind
of weird.

> thank you very much for implementing my suggestion (multiple
> AuthExternal authenticators). I finally had the opportunity to test it
> and it seems to work just fine! I have a question though: there seems
> to be a limit how often one authenticator is tried until it is
> considered to have failed and the next is used. This limit is set to
> 20, is that correct? Is it configurable somewhere?

There isn't any such limit. There is nothing we do 20 times. When there
are multiple authenticators, we try them each once, in order, until one
succeeds.

> I ask because when I have
>
> AuthBasicProvider external
> AuthExternal pwauth mysql-auth
>
> in my Apache configuration each try of the pwauth authenticator seems
> to take about 1-2 seconds to complete (and fail) if the user in
> question is only available through the second authenticator (mysql-
> auth in this case). This results in a delay of about 20-40 seconds
> until the next authenticator is asked. A regular user, unaware of the
> details, would probably think that a network problem or some other
> unexpected error occurred and would likely cancel the whole
> authentication request.
>
> Reversing the order and making mysql-auth the first authenticator
> mitigates that problem since 20 database queries usually don't take
> even a second to complete. The delay is thus not noticeable. But is
> there anything to be done except putting pwauth last? Or did I miss
> something else?

Hmmm...my guess is that you are being hit by pwauth's sleep on bad
logins. I think you'll need to disable it by commenting out SLEEP_LOCK
in config.h .

Here's the comment from config.h:

/* On failed authentications, pwauth will sleep for SLEEP_TIME seconds,
* using a lock on the file whose full path is given by SLEEP_LOCK to
* prevent any other instances of pwauth from running during that time.
* This is meant to slow down password guessing programs, but could be a
* performance problem on extremely heavily used systems. To disable
* this, don't define SLEEP_LOCK. SLEEP_TIME defaults to 2 seconds if
* not defined.
*/

This would also delay the running of the next authenticator, so you
definately don't want this unless you are going to put pwauth last, in
which case it would be no worse an idea than it usually is.

- Jan

[daff]

On Jun 11, 6:16 am, Jan Wolter <j...@unixpapa.com> wrote:
> On Wed, 2009-06-10 at 15:53 -0700, Andreas Ntaflos wrote:
> > Hi Jan (this list seems to have only one -- now two -- member at the
> > moment :)),
>
> Yup.  The list is fairly new.  I'm trying to evolve mod_auth_external
> into a state where it isn't entirely dependent on me to maintain it, and
> provide support for it, so I've shifted to hosting the project on
> code.google.com and added the mailing list.  Not much has come of this
> yet, but that's OK.  I have no plans to stop supporting it myself.
>
> Glad you joined though.  Being the only person on a mailing list is kind
> of weird.

I am glad you keep supporting mod_auth_external! IMHO it should be way
more popular. After all, it is, as far as I can tell, the only
working, non-unmaintained way to authenticate against PAM these days.
The same seems to be true for mod_auth_mysql module which hasn't been
updated since 2005.

> > thank you very much for implementing my suggestion (multiple
> > AuthExternal authenticators). I finally had the opportunity to test it
> > and it seems to work just fine! I have a question though: there seems
> > to be a limit how often one authenticator is tried until it is
> > considered to have failed and the next is used. This limit is set to
> > 20, is that correct? Is it configurable somewhere?
>
> There isn't any such limit.  There is nothing we do 20 times. When there
> are multiple authenticators, we try them each once, in order, until one
> succeeds.

Hm, I probably spoke too soon. I use multiple authenticators to
control access to Subversion repositories from different
authentication databases (PAM and MySQL in my case). I tested this
with the Subversion commandline client and observed those 20 tries
against the first (expected to fail) authenticator. Using a web
browser to access the same repository locations doesn't result in more
than one try on that authenticator (should have tried that first).

So it seems it is the Subversion client that does 20 tries, each
connecting and disconnecting after receiving an authentication
failure, resulting in 20 distinct queries against the first
authenticator. I don't know what makes it eventually stay on long
enough to query the second authenticator but at some point it seems it
happens.

Thanks for the hint. I commented out SLEEP_LOCK to disable pwauth's
waiting SLEEP_TIME seconds and recompiled but that doesn't seem to
have changed the behaviour. So my guess is the Subversion client has
to take all the blame here. I guess I'll keep pwauth as the last
authenticator.

A probably unrelated question but I spotted this in the log files
yesterday while testing: what makes pwauth fail with "-2" and then
segfault?

AuthExtern pwauth [/usr/local/libexec/pwauth]: Failed (-2) for user
foo
user foo: authentication failure for "/some/location/path": Password
Mismatch
External authenticator died on signal 11

This is a location that doesn't have multiple authenticators defined,
only pwauth.

Thanks again!

Andreas

[Jan Wolter]

On Thu, 2009-06-11 at 04:38 -0700, daff wrote:
> I am glad you keep supporting mod_auth_external! IMHO it should be way
> more popular. After all, it is, as far as I can tell, the only
> working, non-unmaintained way to authenticate against PAM these days.
> The same seems to be true for mod_auth_mysql module which hasn't been
> updated since 2005.

I think mod_auth_external is pretty popular, just not so many people
have found the mailing list yet. I think it's easy for these kinds of
modules to fall out of maintanance. Developers tend to build it, use
it, and then move on to other things. Unless someone takes over, it
becomes unmaintained. I'm the third maintainer for mod_auth_external,
and I don't actually administer any production sites that use it any
more. This makes it hard for me to really thoroughly test new versions
and limits my motivation for adding new features. So if you look at the
recent change logs, you'll find most of the changes inspired by someone
else asking for a new feature, or submitting patches for a new feature,
in which case that person is generally available to do some testing.
But really development works much better when the developer is actually
using the software or has close ties to people using the software.

> Hm, I probably spoke too soon. I use multiple authenticators to
> control access to Subversion repositories from different
> authentication databases (PAM and MySQL in my case). I tested this
> with the Subversion commandline client and observed those 20 tries
> against the first (expected to fail) authenticator. Using a web
> browser to access the same repository locations doesn't result in more
> than one try on that authenticator (should have tried that first).

That sounds more and more like a SLEEP_LOCK issue. It's designed to
slow down password guessers. If the subversion client is making 20
consecutive requests with bad passwords in rapid sequence, then that's
exactly the kind of situation where the SLEEP_LOCK is supposed to dig in
it's heels and slow things down.

OK, I dug through the code, and I see that undefining SLEEP_LOCK doesn't
actually completely turn of the sleep delay. I think that counts as a
bug. I've attached a new version of one of the source code files for
pwauth, sleep.c. Try substituting that in and compiling it with
SLEEP_LOCK undefined, and see if this behavior improves.

> Thanks for the hint. I commented out SLEEP_LOCK to disable pwauth's
> waiting SLEEP_TIME seconds and recompiled but that doesn't seem to
> have changed the behaviour. So my guess is the Subversion client has
> to take all the blame here. I guess I'll keep pwauth as the last
> authenticator.

I still think pwauth is partly to blame here.

> A probably unrelated question but I spotted this in the log files
> yesterday while testing: what makes pwauth fail with "-2" and then
> segfault?
>
> AuthExtern pwauth [/usr/local/libexec/pwauth]: Failed (-2) for user
> foo
> user foo: authentication failure for "/some/location/path": Password
> Mismatch
> External authenticator died on signal 11
>
> This is a location that doesn't have multiple authenticators defined,
> only pwauth.

Hmmm...those messages seem out of order. The "died on signal 11" should
come first, then the other two, but that probably doesn't matter.

All that can be told from this is that pwauth seems to have seg-faulted.
The other two messages are just aftershocks from that rattling through
the system and given no additional information.

Obviously anytime any programs seg faults, that's a bug. Since seg
faults have to do with incorrectly addressing memory, that could even be
a sign of a potential security problem. Do I understand correctly that
you are using it in PAM mode? Any details you can give me about what
circumstances it is segfaulting under would be of tremendous interest.

- Jan

[daff]

Thanks, I applied the patch and now undefining SLEEP_LOCK works as
expected. The 20 tries against pwauth now complete much faster.

> > Thanks for the hint. I commented out SLEEP_LOCK to disable pwauth's
> > waiting SLEEP_TIME seconds and recompiled but that doesn't seem to
> > have changed the behaviour. So my guess is the Subversion client has
> > to take all the blame here. I guess I'll keep pwauth as the last
> > authenticator.
>
> I still think pwauth is partly to blame here.

Do you think there can be anything else done except disabling pwauth's
sleeping?

> > A probably unrelated question but I spotted this in the log files
> > yesterday while testing: what makes pwauth fail with "-2" and then
> > segfault?
>
> > AuthExtern pwauth [/usr/local/libexec/pwauth]: Failed (-2) for user
> > foo
> > user foo: authentication failure for "/some/location/path": Password
> > Mismatch
> > External authenticator died on signal 11
>
> > This is a location that doesn't have multiple authenticators defined,
> > only pwauth.
>
> Hmmm...those messages seem out of order.  The "died on signal 11" should
> come first, then the other two, but that probably doesn't matter.

I thought so, too. But it is this way consistently throughout the log
files.

> All that can be told from this is that pwauth seems to have seg-faulted.
> The other two messages are just aftershocks from that rattling through
> the system and given no additional information.
>
> Obviously anytime any programs seg faults, that's a bug.  Since seg
> faults have to do with incorrectly addressing memory, that could even be
> a sign of a potential security problem.  Do I understand correctly that
> you are using it in PAM mode?  Any details you can give me about what
> circumstances it is segfaulting under would be of tremendous interest.

I am using PAM, yes, as in "#define PAM" in config.h. Other than that
I changed the SERVER_UIDS to 33 (www-data on Debian) and now undefined
SLEEP_LOCK. I also have UNIX_LASTLOG and HAVE_LASTLOG_H defined

#define UNIX_LASTLOG /**/
#define HAVE_LASTLOG_H /**/

which I now realise I shouldn't when using PAM, right? Could this be a
problem?

The thing is that those segfaults occur only with one user ('foo' as I
called him) on exactly two locations; both are timelines of Trac
installations as in [1]. I asked that user to provide more details and
will get back to you as soon as I know anything. It seems though that
it is some kind of RSS reader/bot that accesses these locations.

[1] http://trac.edgewall.org/timeline