authentication problem for one ldap user (out of hundreds) because external ID is already in use
132 views
Skip to first unread message
euphxenos
Nov 20, 2019, 3:26:52 PM11/20/19
to Repo and Gerrit Discussion
We use LDAP for authentication. It's the only authentication method we've used so far. I have one user who is unable to login. When this user tries to log in, they get "Invalid username or password." in the web ui. I'm reasonably sure that the user is entering their correct password, because I see different error messages go to error_log depending on whether or not we give the correct password or a deliberately wrong password. What I see in the error_log when the user gives the right password is:
[2019-11-19 01:25:20,881] [HTTP-58336] WARN com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'vpemmaka' failed to sign in
com.google.gerrit.server.account.AccountException: Cannot assign external ID "username:vpemmaka" to account 487; external ID already in use.
followed by a backtrace (complete error included below). The user is already in All-Users as account 417. Every time the user tries to log i, we get this error with another new account id. It looks like Gerrit doesn't recognize that the user is already there, then tries to create a new account in All-Users, *then* realizes the external ID is already in use, and fails. I've filed a bug for this (https://bugs.chromium.org/p/gerrit/issues/detail?id=11960), but my pressing need is to figure out how I can clean up this user's account so they can log in. We're running 2.16.8. We converted from reviewdb to notedb when we upgraded from 2.15.x to 2.16.8. I've also tested that 3.0.4 behaves the same way (I've cloned the vm Gerrit runs in to create a test environment, upgraded the clone, and had the user try to login there as well). When I look at this user's account using admin-console show-account, the thing I notice that looks different is that this user has two external ids listed, a username: and mailto:, where all the other accounts have gerrit: and username: external ids. I have checked to make sure this really is the same user. We do not have multiple LDAP users with the same aAMAccountName, and this is not a recycled sAMAccountName from a different person. This user's account was created in gerrit before we converted from reviewdb to notedb.
I suspect that this can probably be corrected by manually adjusting the account metadata in All-Users, replacing his mailto: external id with a gerrit: external id. Does this sound like a reasonable approach, and if so, what exactly do I need to do? I'm poking around in All-Users, and have found the external ids in refs/heads/external-ids. I see that each user appears in two different files, the filenames appear to be shas, and it looks like there's some sort of sharding going on here. If I just change the contents of the file with his mailto: external id to contain a gerrit: external id instead, is that sufficient or will that break whatever indexing is in use? Would I need to move that external id file to another location because of the change in contents? Is there some way to run a consistency check on All-Users? Am I going in completely the wrong direction?
thanks,
--Andrew
[2019-11-19 01:25:20,881] [HTTP-58336] WARN com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'vpemmaka' failed to sign in
com.google.gerrit.server.account.AccountException: Cannot assign external ID "username:vpemmaka" to account 487; external ID already in use.
at com.google.gerrit.server.account.AccountManager.create(AccountManager.java:306)
at com.google.gerrit.server.account.AccountManager.authenticate(AccountManager.java:140)
at com.google.gerrit.httpd.auth.ldap.LdapLoginServlet.doPost(LdapLoginServlet.java:123)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:453)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.UniversalWebLoginFilter.doFilter(UniversalWebLoginFilter.java:74)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:121)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.GwtCacheControlFilter.doFilter(GwtCacheControlFilter.java:72)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:129)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:69)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Unknown Source)
Luca Milanesio
Nov 20, 2019, 3:57:38 PM11/20/19
to euphxenos, Luca Milanesio, Repo and Gerrit Discussion
Hi Andrew,
see my comments inline.
On 20 Nov 2019, at 20:26, euphxenos <euph...@gmail.com> wrote:We use LDAP for authentication. It's the only authentication method we've used so far. I have one user who is unable to login. When this user tries to log in, they get "Invalid username or password." in the web ui. I'm reasonably sure that the user is entering their correct password, because I see different error messages go to error_log depending on whether or not we give the correct password or a deliberately wrong password. What I see in the error_log when the user gives the right password is:[2019-11-19 01:25:20,881] [HTTP-58336] WARN com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'vpemmaka' failed to sign incom.google.gerrit.server.account.AccountException: Cannot assign external ID "username:vpemmaka" to account 487; external ID already in use.followed by a backtrace (complete error included below). The user is already in All-Users as account 417. Every time the user tries to log i, we get this error with another new account id. It looks like Gerrit doesn't recognize that the user is already there, then tries to create a new account in All-Users, *then* realizes the external ID is already in use, and fails. I've filed a bug for this (https://bugs.chromium.org/p/gerrit/issues/detail?id=11960), but my pressing need is to figure out how I can clean up this user's account so they can log in. We're running 2.16.8.
Gerrit v2.16.8 is pretty behind the latest bug-fix release, which is v2.16.13.
You should always stay up-to-date with the latest bug-fix release, because it may contain exactly the fixes you need.
For example, if you look at the v2.16.13 release notes (see [1]), you'll see:
"Issue 11246: Fix login failures “email <> is already assigned to account <> …” after upgrading to v2.16."
HTH
Luca.
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/66cb3cb3-1140-4318-9420-1a79fe8c5f53%40googlegroups.com.
euphxenos
Nov 20, 2019, 4:15:25 PM11/20/19
to Repo and Gerrit Discussion
Hi Luca,
Response inline
On Wednesday, November 20, 2019 at 12:57:38 PM UTC-8, lucamilanesio wrote:
Hi Andrew,see my comments inline.On 20 Nov 2019, at 20:26, euphxenos <euph...@gmail.com> wrote:We use LDAP for authentication. It's the only authentication method we've used so far. I have one user who is unable to login. When this user tries to log in, they get "Invalid username or password." in the web ui. I'm reasonably sure that the user is entering their correct password, because I see different error messages go to error_log depending on whether or not we give the correct password or a deliberately wrong password. What I see in the error_log when the user gives the right password is:[2019-11-19 01:25:20,881] [HTTP-58336] WARN com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'vpemmaka' failed to sign incom.google.gerrit.server.account.AccountException: Cannot assign external ID "username:vpemmaka" to account 487; external ID already in use.followed by a backtrace (complete error included below). The user is already in All-Users as account 417. Every time the user tries to log i, we get this error with another new account id. It looks like Gerrit doesn't recognize that the user is already there, then tries to create a new account in All-Users, *then* realizes the external ID is already in use, and fails. I've filed a bug for this (https://bugs.chromium.org/p/gerrit/issues/detail?id=11960), but my pressing need is to figure out how I can clean up this user's account so they can log in. We're running 2.16.8.Gerrit v2.16.8 is pretty behind the latest bug-fix release, which is v2.16.13.You should always stay up-to-date with the latest bug-fix release, because it may contain exactly the fixes you need.For example, if you look at the v2.16.13 release notes (see [1]), you'll see:"Issue 11246: Fix login failures “email <> is already assigned to account <> …” after upgrading to v2.16."
I appreciate that 2.16.8 is behind, but as part of an upcoming upgrade I also tried cloning our 2.16.8 Gerrit VM, upgrading the clone to 3.0.4 (which I believe is the current 3.0 release), and having the user try logging into the upgraded system, with the same results. In other words, I'm seeing this problem on both 2.16.8 (in production) and after upgrading to 3.0.4 (in my upgrade staging environment). Issue 11246 claims to be fixed in 3.0.3, so unless 3.0.4 regressed that bug fix, I would expect 3.0.4 to be sufficient to pick that up. Am I missing something here?
thanks,
--Andrew
To unsubscribe, email rep...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.
Luca Milanesio
Nov 20, 2019, 4:36:35 PM11/20/19
to euphxenos, Luca Milanesio, Repo and Gerrit Discussion
On 20 Nov 2019, at 21:15, euphxenos <euph...@gmail.com> wrote:Hi Luca,Response inline
On Wednesday, November 20, 2019 at 12:57:38 PM UTC-8, lucamilanesio wrote:Hi Andrew,see my comments inline.On 20 Nov 2019, at 20:26, euphxenos <euph...@gmail.com> wrote:We use LDAP for authentication. It's the only authentication method we've used so far. I have one user who is unable to login. When this user tries to log in, they get "Invalid username or password." in the web ui. I'm reasonably sure that the user is entering their correct password, because I see different error messages go to error_log depending on whether or not we give the correct password or a deliberately wrong password. What I see in the error_log when the user gives the right password is:[2019-11-19 01:25:20,881] [HTTP-58336] WARN com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'vpemmaka' failed to sign incom.google.gerrit.server.account.AccountException: Cannot assign external ID "username:vpemmaka" to account 487; external ID already in use.followed by a backtrace (complete error included below). The user is already in All-Users as account 417. Every time the user tries to log i, we get this error with another new account id. It looks like Gerrit doesn't recognize that the user is already there, then tries to create a new account in All-Users, *then* realizes the external ID is already in use, and fails. I've filed a bug for this (https://bugs.chromium.org/p/gerrit/issues/detail?id=11960), but my pressing need is to figure out how I can clean up this user's account so they can log in. We're running 2.16.8.Gerrit v2.16.8 is pretty behind the latest bug-fix release, which is v2.16.13.You should always stay up-to-date with the latest bug-fix release, because it may contain exactly the fixes you need.For example, if you look at the v2.16.13 release notes (see [1]), you'll see:"Issue 11246: Fix login failures “email <> is already assigned to account <> …” after upgrading to v2.16."
Actually, the fix is about the email external id duplicates, not the username.
What I believe is happening, is that for some reason the user login triggers an account creation, instead of locating the existing account.
What's the situation of this account external ids?
You may have found another bug that we haven't fixed yet.
Luca.
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0b4042f9-911d-488e-a61a-c1e2d48be91b%40googlegroups.com.
euphxenos
Nov 20, 2019, 4:50:52 PM11/20/19
to Repo and Gerrit Discussion
On Wednesday, November 20, 2019 at 1:36:35 PM UTC-8, lucamilanesio wrote:
On 20 Nov 2019, at 21:15, euphxenos <euph...@gmail.com> wrote:Hi Luca,Response inline
On Wednesday, November 20, 2019 at 12:57:38 PM UTC-8, lucamilanesio wrote:Hi Andrew,see my comments inline.On 20 Nov 2019, at 20:26, euphxenos <euph...@gmail.com> wrote:We use LDAP for authentication. It's the only authentication method we've used so far. I have one user who is unable to login. When this user tries to log in, they get "Invalid username or password." in the web ui. I'm reasonably sure that the user is entering their correct password, because I see different error messages go to error_log depending on whether or not we give the correct password or a deliberately wrong password. What I see in the error_log when the user gives the right password is:[2019-11-19 01:25:20,881] [HTTP-58336] WARN com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'vpemmaka' failed to sign incom.google.gerrit.server.account.AccountException: Cannot assign external ID "username:vpemmaka" to account 487; external ID already in use.followed by a backtrace (complete error included below). The user is already in All-Users as account 417. Every time the user tries to log i, we get this error with another new account id. It looks like Gerrit doesn't recognize that the user is already there, then tries to create a new account in All-Users, *then* realizes the external ID is already in use, and fails. I've filed a bug for this (https://bugs.chromium.org/p/gerrit/issues/detail?id=11960), but my pressing need is to figure out how I can clean up this user's account so they can log in. We're running 2.16.8.Gerrit v2.16.8 is pretty behind the latest bug-fix release, which is v2.16.13.You should always stay up-to-date with the latest bug-fix release, because it may contain exactly the fixes you need.For example, if you look at the v2.16.13 release notes (see [1]), you'll see:"Issue 11246: Fix login failures “email <> is already assigned to account <> …” after upgrading to v2.16."Actually, the fix is about the email external id duplicates, not the username.What I believe is happening, is that for some reason the user login triggers an account creation, instead of locating the existing account.
Yes, that's my reading of the error message. Each time the user tries to log in, the error reports that it's unable to assign the external id to a new account id, because that external id is already in use. After that, I see the new account id does not exist in All-Users. If the user tries again, the error message repeats with another new account id in the error message.
What's the situation of this account external ids?
I'm not entirely sure what you're asking. In refs/heads/external-ids I see two entries for this user:
f1/aa6477e6be17cdf9d11d1aafc209a6efef5c75:
[externalId "username:vpemmaka"]
accountId = 417
password = <expunged for this post>
39/e877de61336c67d6357dc577ae3e0e9521a921:
[externalId "mailto:<expunged his email address>"]
accountId = 417
email = <expunged his email address>
I do *not* see an entry in refs/heads/external-ids for this user that includes a gerrit:<username> as the external id, which I do see for every other account I've looked at. Does that answer your question? Will a user be able to log in without a gerrit: entry in refs/heads/external-ids?
thanks,
--Andrew
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0b4042f9-911d-488e-a61a-c1e2d48be91b%40googlegroups.com.
Andrew Grimberg
Nov 21, 2019, 1:35:49 PM11/21/19
to euphxenos, Repo and Gerrit Discussion
On 11/20/19 1:50 PM, euphxenos wrote:
>
>
> On Wednesday, November 20, 2019 at 1:36:35 PM UTC-8, lucamilanesio wrote:
--[snip]--
>
>
> On Wednesday, November 20, 2019 at 1:36:35 PM UTC-8, lucamilanesio wrote:
> What's the situation of this account external ids?
>
>
> I'm not entirely sure what you're asking. In refs/heads/external-ids I
> see two entries for this user:
>
> f1/aa6477e6be17cdf9d11d1aafc209a6efef5c75:
>
> [externalId "username:vpemmaka"]
> accountId = 417
> password = <expunged for this post>
>
> 39/e877de61336c67d6357dc577ae3e0e9521a921:
>
> [externalId "mailto:<expunged his email address>"]
> accountId = 417
> email = <expunged his email address>
>
> I do *not* see an entry in refs/heads/external-ids for this user that
> includes a gerrit:<username> as the external id, which I do see for
> every other account I've looked at. Does that answer your question?
> Will a user be able to log in without a gerrit: entry in
> refs/heads/external-ids?
may have lost the entry if they had registered a new email, set it as
their primary, and then deleted the email address that is in your LDAP.
I've seen this happen before.
To recreate the missing entry you need to do the following:
echo -n 'gerrit:<username>' | sha1sum
cp 39/e877de61336c67d6357dc577ae3e0e9521a921 <the_sha1>
NOTE: The sha1 you got is going to need to be sharded by adding a '/'
after the first two characters in it.
for instance:
--[cut]--
echo -n 'gerrit:agrimberg' | sha1sum
735fb2e607b5783897bc3d301f4c9c8be0b2959d -
--[/cut]--
Therefore the resultant location would be:
73/5fb2e607b5783897bc3d301f4c9c8be0b2959d
After that, edit the file and change out the "mailto:..." to the
'gerrit:...' and make sure the email is set to the same one as is in
LDAP. If you don't do this you're going to have problems.
Then push that back up:
git push origin HEAD:refs/meta/external-ids
-Andy-
euphxenos
Nov 27, 2019, 3:16:42 PM11/27/19
to Repo and Gerrit Discussion
Thanks; this was exactly what I was looking for. Between the upgrade to 3.0.4 *and* this change to the user's external-ids in All-Users, the user is able to log in again. Just the LDAP fix in 3.0.4 wasn't enough, but this metadata fix is rejected by Gerrit if I try to apply it to 2.16.8. I suspect I could make that work by removing the mailto external id, but it sounds like it could then easily be broken again by the user.
Reconstructing how this happened, the user's LDAP account had their email address listed wrong in the mail field in their account. When the user initially logged into Gerrit, Gerrit created a local account, but email to the user failed because the address was wrong, so the user updated their email address in Gerrit (and only in Gerrit, leaving it wrong in LDAP). At that point the user couldn't log in. We got the IT folks who manage the LDAP server to correct the user's email address in LDAP, then we could apply this fix on the Gerrit side to get the user back in.
It seems like there's room to do more on the Gerrit side for users impacted by this. If it can't automatically be corrected, perhaps at least tell the administrator which accounts are corrupted in such a way that it's impossible for the user to log in? I'm not sure where that would even go in the UI, since there's no account management in the web UI at all today (which is also a problem).
--Andrew
Reply all
Reply to author
Forward
0 new messages