user can't log in after email change after upgrade 2.12 -> 2.16

[Jiri Tomek]

Hi,

last week I did upgrade Gerrit from 2.12.2 ->  2.13.12 -> 2.14.18 -> 2.15.9 -> 2.16.4. For each version I did init and reindex. So far the upgrade went OK.

Our Gerrit installation is connected to LDAP.

Next day one of our developers has changed his email in Gerrit but didn't requested to change it also in LDAP where was still his old email. When he attempted to log in to Gerrit he got "authentication failed".

In the error log I can see this:

[2019-02-08 14:00:21,547] [HTTP-2408] WARN  com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'bancrom' failed to sign in

com.google.gerrit.server.account.AccountException: Cannot assign external ID "username:bancrom" to account 124; external ID already in use.

        at com.google.gerrit.server.account.AccountManager.create(AccountManager.java:306)

        at com.google.gerrit.server.account.AccountManager.authenticate(AccountManager.java:140)

        at com.google.gerrit.httpd.auth.ldap.LdapLoginServlet.doPost(LdapLoginServlet.java:123)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)

        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)

        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)

        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

        at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:485)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.UniversalWebLoginFilter.doFilter(UniversalWebLoginFilter.java:74)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:121)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.GwtCacheControlFilter.doFilter(GwtCacheControlFilter.java:72)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133)

        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:69)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)

        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)

        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)

        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)

        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)

        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

        at org.eclipse.jetty.server.Server.handle(Server.java:503)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)

        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)

        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)

        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)

        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)

        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)

        at java.lang.Thread.run(Thread.java:748)

I realized the emails are different so I changed the to be the same on Gerrit and LDAP. But this didn't help and Gerrit still  wants to create a new account. Now the error message is:

[2019-02-13 14:38:49,429] [HTTP-320] WARN  com.google.gerrit.server.account.AccountManager : Email ban...@ourcompany.net is already assigned to account 21; cannot create external ID gerrit:bancrom with the same email for account 173.

[2019-02-13 14:38:49,429] [HTTP-320] WARN  com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'bancrom' failed to sign in

com.google.gerrit.server.account.AccountException: Email 'ban...@ourcompany.net' in use by another account

at com.google.gerrit.server.account.AccountManager.checkEmailNotUsed(AccountManager.java:374)

at com.google.gerrit.server.account.AccountManager.create(AccountManager.java:281)

at com.google.gerrit.server.account.AccountManager.authenticate(AccountManager.java:140)

at com.google.gerrit.httpd.auth.ldap.LdapLoginServlet.doPost(LdapLoginServlet.java:123)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:485)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.UniversalWebLoginFilter.doFilter(UniversalWebLoginFilter.java:74)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:121)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.GwtCacheControlFilter.doFilter(GwtCacheControlFilter.java:72)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:69)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

at org.eclipse.jetty.server.Server.handle(Server.java:503)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)

at java.lang.Thread.run(Thread.java:748)

I appreciate any help with this. 

Best regards

Jiri

[Jiri Tomek]

Does anybody have a clue why Gerrit wants to create a new account? Is there any possibility to force Gerrit to use the old account and not create new one? 

[Karl]

We exactly have the same problem... so far I cannot find any solution/workarounds for this issue.

I upgraded our LDAP connected gerrit server from 2.15.x to 2.16.5. 

I also migrated fully (including changes) from ReviewDb to NoteDb according to https://www.gerritcodereview.com/2.16.html#migration-to-notedb

As a result, users who were registered after upgrade cannot login at all.

Here is our error log:

[2019-02-27 14:56:12,838] [HTTP-5170] WARN  com.google.gerrit.server.account.AccountManager : Email user.registered.a...@example.net is already assigned to account GERRIT_USER_ID; cannot create external ID gerrit:sAMAccountName-for-this-user with the same email for account GERRIT_USER_ID_GERRIT_SOMEHOW_TRIED_TO_CREATE.

[2019-02-27 14:56:12,838] [HTTP-5170] WARN  com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'sAMAccountName-for-this-user' failed to sign in

com.google.gerrit.server.account.AccountException: Email 'user.registere...@example.net' in use by another account

[Karl]

In our case, there are a difference in All-Users.git. 

$ mkdir ~/All-Users.nonbare

$ cp -r /data/repos/All-Users.git ~/All-Users.nonbare/.git

$ cd ~/All-Users.nonbare

$ git config core.bare false

$ git checkout meta/external-ids

a) Users who can sign in

$ grep -ilr 'user-can...@example.net' ./

.//9e/SHA1_HASH_1

$ cat .//9e/SHA1_HASH_1

[externalId "gerrit:sAMAccountName1"]

    accountId = 1111111

    email = user-can...@example.net

b) Users who cannot sign in

$ grep -ilr 'user-cann...@example.net' ./

.//4b/SHA_HASH_2

$ cat .//4b/SHA_HASH_2

[externalId "mailto:user-cann...@example.net"]

    accountId = 2222222

    email = user-cann...@example.net

I'll delete 'mailto:' records and check whether they can sign in or not. 

[anders.ha...@gmail.com]

We also have this problem with few accounts.  

[anders.ha...@gmail.com]

We now have two users that can't login to our Gerrit instance.

It appears that there was no externalId gerrit:<id> create the first time they logged in.

The rest of the account detauls are there including an mailto: externalId.

This prevents the user from logging in, Gerrit is complaining that the emailaddress and username is already in-use.

I also tried to manually create the external id entry in All-Users but I'm unable to push the change.

Gerrit is complaining that lot of emailaddress are not unique.

Our Gerrit is using ldap auth.

/Anders

[anders.ha...@gmail.com]

Managed to find a "workaround" by manually manipulating the users external-ids in the All-Users repo.

[Karl]

I've found a workaround in only case of registering new user. 

Firstly, if I registered an user with options, gerrit inserts 2 records. One is users/yy/xxxxxyy, the other is meta/external-ids for users/yy/xxxxxyy. 

meta/external-ids for users/yy/xxxxxyy causes an exception.

However, if I registered an user with username(in our case, sAMAccount) only, gerrit only inserts 1 records - users/yy/xxxxxyy. 

Still, gerrit tries to create an new account when users sign in firstly, but gerrit doesn't throw an exception and users can sign in. 

Because there are no confliction in meta/external-ids. 

(Of course, I have to do granting accesses and etc for such users after that. It's extra work compared with the past...)

Remaining concern is - in case of existing users as same as Jiri and Anders. In our case, there will be massive changes in LDAP user data soon due to many transfers in the beginning next fiscal year. 

Do I need to manipulate manually All-Users.git for them?

It's nightmare :-(

[Karl]

Try the following: 

$ sudo cp -rp /PATH_TO_REPOSITORIES/All-Users.git /PATH_TO_REPOSITORIES/All-Users.git.bak

$ mkdir ~/All-Users.nonbare

$ sudo cp -r /PATH_TO_REPOSITORIES/All-Users.git ~/All-Users.nonbare/.git

$ cd ~/All-Users.nonbare

$ git config core.bare false

$ git checkout refs/meta/external-ids

$ grep -ilr "GERRIT_USERNAME_WHO_CANNOT_LOGIN" ./|xargs rm

$ git remote add origin /PATH_TO_REPOSITORIES/All-Users.git

$ git rm $(git ls-files --deleted)

$ git commit -m "Delete external ids from users who cannot sign in."

$ git push origin HEAD:refs/meta/external-ids

Note:

Once you confirmed he/she could sign in, you might need to change settings for him/her due to user id will be changed. 

2019年2月18日月曜日 16時35分06秒 UTC+9 Jiri Tomek:

[Krešimir Tonković]

I was hit by the same issue - thanks for the great hint Karl.

Once I checked out refs/meta/external-ids I was able to look at the log and I found the change where the affected user Unlinked his external ID. I reverted that change and pushed it back to the bare repo and the user was able to work again.

it is still not 100% clear how the user was able to unlink his account. By his words, he only changed his preferred email address after registering a new one.

Considering you'll be pushing to the remote directly rather than through gerrit, make sure you're doing that as the gerrit user. Otherwise gerrit may not be able to read the updated file due to file system permissions.

Not sure if it is needed, I also reindexed the user in question to make sure gerrit knew about his latest state:

curl -X POST -u <your user>:<your http pass> https://<gerrit host>/a/accounts/<affected user id>/index

--

Regards,

Kresimir

[Edwin Kempin]

On Wed, Apr 17, 2019 at 12:49 PM Krešimir Tonković <kresimir...@gmail.com> wrote:

I was hit by the same issue - thanks for the great hint Karl.

Once I checked out refs/meta/external-ids I was able to look at the log and I found the change where the affected user Unlinked his external ID. I reverted that change and pushed it back to the bare repo and the user was able to work again.

it is still not 100% clear how the user was able to unlink his account. By his words, he only changed his preferred email address after registering a new one.

The UI allows users to delete their identities. Maybe that was done by mistake.

 

Considering you'll be pushing to the remote directly rather than through gerrit, make sure you're doing that as the gerrit user. Otherwise gerrit may not be able to read the updated file due to file system permissions.

Not sure if it is needed, I also reindexed the user in question to make sure gerrit knew about his latest state:

curl -X POST -u <your user>:<your http pass> https://<gerrit host>/a/accounts/<affected user id>/index

--

Regards,

Kresimir

On Wednesday, March 6, 2019 at 6:41:11 AM UTC+1, Karl wrote:

Try the following: 

$ sudo cp -rp /PATH_TO_REPOSITORIES/All-Users.git /PATH_TO_REPOSITORIES/All-Users.git.bak

$ mkdir ~/All-Users.nonbare

$ sudo cp -r /PATH_TO_REPOSITORIES/All-Users.git ~/All-Users.nonbare/.git

$ cd ~/All-Users.nonbare

$ git config core.bare false

$ git checkout refs/meta/external-ids

$ grep -ilr "GERRIT_USERNAME_WHO_CANNOT_LOGIN" ./|xargs rm

$ git remote add origin /PATH_TO_REPOSITORIES/All-Users.git

$ git rm $(git ls-files --deleted)

$ git commit -m "Delete external ids from users who cannot sign in."

$ git push origin HEAD:refs/meta/external-ids

Note:

Once you confirmed he/she could sign in, you might need to change settings for him/her due to user id will be changed. 

2019年2月18日月曜日 16時35分06秒 UTC+9 Jiri Tomek:

Does anybody have a clue why Gerrit wants to create a new account? Is there any possibility to force Gerrit to use the old account and not create new one? 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Makson Lee]

we also have one account which don't have the gerrit:<id> external id, how did you fix that?

[Doug Luedtke]

On Tuesday, March 5, 2019 at 11:41:11 PM UTC-6, Karl wrote:

Try the following: 

$ sudo cp -rp /PATH_TO_REPOSITORIES/All-Users.git /PATH_TO_REPOSITORIES/All-Users.git.bak

$ mkdir ~/All-Users.nonbare

$ sudo cp -r /PATH_TO_REPOSITORIES/All-Users.git ~/All-Users.nonbare/.git

$ cd ~/All-Users.nonbare

$ git config core.bare false

$ git checkout refs/meta/external-ids

$ grep -ilr "GERRIT_USERNAME_WHO_CANNOT_LOGIN" ./|xargs rm

$ git remote add origin /PATH_TO_REPOSITORIES/All-Users.git

$ git rm $(git ls-files --deleted)

$ git commit -m "Delete external ids from users who cannot sign in."

$ git push origin HEAD:refs/meta/external-ids

Was the push done while Gerrit was still online?

[Doug Luedtke]

On Wednesday, April 17, 2019 at 5:49:09 AM UTC-5, Krešimir Tonković wrote:

I was hit by the same issue - thanks for the great hint Karl.

Once I checked out refs/meta/external-ids I was able to look at the log and I found the change where the affected user Unlinked his external ID. I reverted that change and pushed it back to the bare repo and the user was able to work again.

it is still not 100% clear how the user was able to unlink his account. By his words, he only changed his preferred email address after registering a new one.

Considering you'll be pushing to the remote directly rather than through gerrit, make sure you're doing that as the gerrit user. Otherwise gerrit may not be able to read the updated file due to file system permissions.

Not sure if it is needed, I also reindexed the user in question to make sure gerrit knew about his latest state:

curl -X POST -u <your user>:<your http pass> https://<gerrit host>/a/accounts/<affected user id>/index

Was the push done while Gerrit was online?

[Krešimir Tonković]

Can't remember, but I'm sure it'd be safer if it was offline.

Regards,

Kresimir

[Doug Luedtke]

On Thursday, October 24, 2019 at 12:45:19 AM UTC-5, Krešimir Tonković wrote:

Can't remember, but I'm sure it'd be safer if it was offline.

Regards,

Kresimir

Thank you. 

I went ahead and took a lot of time testing yesterday. I used the application's user to do the commit. All done while Gerrit was running. The refs/meta/external-ids branch doesn't get written to often, so that is why I decided to test it that way. Also, taking an outage requires coordinating with multiple teams with offices covering all timezones.

It worked in test, it worked on a quiet master, and then it worked on a busy master. I'll upgrade to Gerrit 2.16.12 hopefully soon to not have to do that again. I'm on 2.16.10 at the moment.