Login issue after email address assignment in LDAP/AD (Email <email> is already assigned to account 1000027; cannot create external ID gerrit:<username> with the same email for account 1000027.,)

[Michael Croes]

Hi list,

Today the domain admins decided to assign email addresses to user accounts which previously didn't have them assigned. As a result none of the users that previously didn't have an address set can log in. I'm seeing the following errors in the log:

[2019-09-24 13:52:44,053] [HTTP-142] WARN  com.google.gerrit.server.account.AccountManager : Email <email> is already assigned to account 1000027; cannot create external ID gerrit:<username> with the same email for account 1000027.,

[2019-09-24 13:52:44,053] [HTTP-142] WARN  com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : '<username>' failed to sign in

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742),

at com.google.gerrit.server.auth.ldap.Helper.findAccount(Helper.java:268),

at com.google.gerrit.server.auth.ldap.LdapRealm.authenticate(LdapRealm.java:242),

at com.google.gerrit.server.account.AccountManager.authenticate(AccountManager.java:131),

at com.google.gerrit.httpd.auth.ldap.LdapLoginServlet.doPost(LdapLoginServlet.java:123),

at javax.servlet.http.HttpServlet.service(HttpServlet.java:661),

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290),

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280),

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184),

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89),

at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:439),

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85),

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82),

at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92),

at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72),

at com.google.gerrit.httpd.UniversalWebLoginFilter.doFilter(UniversalWebLoginFilter.java:75),

at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120),

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133),

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135),

at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57),

at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:69),

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121),

at java.lang.Thread.run(Thread.java:748),

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540),

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133),

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610),

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588),

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203),

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557),

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201),

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310),

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260),

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144),

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247),

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345),

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480),

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255),

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132),

at org.eclipse.jetty.server.Server.handle(Server.java:502),

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364),

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305),

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118),

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103),

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333),

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168),

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126),

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366),

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765),

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)

All these users were able to login before the email addresses got assigned. I've been trying to figure out why Gerrit has issues with this. I did discover that in All-Users:refs/meta/external-ids there's an address set for my own account in the file that has externalId "gerrit:<ny-username>", but it's missing for those that previously didn't have the email address set.

I hope anyone can advise me on this issue.

Kind regards,

Michael

[Michael Croes]

Hi list,

Thanks for the rubber duck therapy. I found that there was a behavior that had changed in Gerrit 2.16, requiring an email address to be unique across all identities. Somehow that wasn't the case, because in external-ids I also had externalId "mailto:<email>" for all those accounts that were facing issues. A simple grep -l mailto * | xargs git rm in refs/meta/external-ids and pushing that fixed the issue entirely. Now when the users login the email gets added to the externalId "gerrit:<username>" without issues.

Kind regards,

Michael

[Luca Milanesio]

On 24 Sep 2019, at 15:48, Michael Croes <myc...@gmail.com> wrote:

Hi list,

Thanks for the rubber duck therapy. I found that there was a behavior that had changed in Gerrit 2.16, requiring an email address to be unique across all identities.

Hi Michael,

I just tried with Gerrit v2.15 and I confirm that when I want to use the e-mail 'jo...@gerritforge.com' in two accounts I get:

"Code Review - Error

Identity 'mailto:jo...@gerritforge.com' in use by another account"

AFAIK, the external-ids must be unique IDs across all gerrit:

- Two accounts cannot share the same username

- Two accounts cannot share the same e-mail

It has been like that for a *veery long* time :-)

I believe you are hitting a different issue, possibly related to [1].

After the migration, some users cannot be authenticated and thus Gerrit try to re-create them. However, the creation fails because the LDAP e-mail already exists as external id in Gerrit.

The fix [1] was working (it was actually a workaround), however, it was reverted with [2] because of security issues.

HTH

[1] https://gerrit-review.googlesource.com/c/gerrit/+/162450

[2] https://gerrit-review.googlesource.com/c/gerrit/+/209089

Luca.

[Michael Croes]

Hi Luca,

The externalIds were unique, but for every account I had gerrit:<username> and mailto:<email> externalIds. Obviously the mailto externalId had an email set, and for some accounts there also was an email set on the gerrit:<user> externalId. I actually couldn't even push refs/meta/external-ids as it was in Gerrit due to the duplicate email values.

So I have to disagree with you, nevertheless thanks for your response.

Regards,
Michael

[Luca Milanesio]

On 24 Sep 2019, at 16:36, Michael Croes <myc...@gmail.com> wrote:

Hi Luca,

The externalIds were unique, but for every account I had gerrit:<username> and mailto:<email> externalIds. Obviously the mailto externalId had an email set, and for some accounts there also was an email set on the gerrit:<user> externalId.

Ah, I do understand now.

I believe we should invest more in the "init step" and detect those situation during upgrade.

It is very unpleasant finding out *after* the upgrade that people cannot login anymore.

I do understand and share your pain.

I am preparing a talk at the next forthcoming Gerrit User Summit in Sunnyvale (gus2019.eventbrite.com) and I will definitely mention the "consistency check" step.

I'll try also to write some tools to help people migrating and detect those issues early on.

I actually couldn't even push refs/meta/external-ids as it was in Gerrit due to the duplicate email values.

Yes, you should sort out those issues on the server side, by pushing directly to the repo bypassing Gerrit in that case.

So I have to disagree with you, nevertheless thanks for your response.

It's good to disagree at times, as long as we keep talking :-)

And thanks for your feedback, much appreciated.

Luca.

[Andrew Grimberg]

I posted a process here [0] on July 26 on how to essentially work
through this particular problem. I've got a personal extension to it now
on how to merge accounts too, but I'm not going to go into that.

I strongly believe that the 'gerrit:<username>' object should _not_ have
a mail string in it and all mail links in meta/external-id should be
'mailto:<email>' records. This would resolve the problem of your
external identity provider changing the email address.

Especially since email addresses change _very_ regularly! Not as
regularly in a corporate environment, but if you're running a public
facing Gerrit it happens all the darn time!

I (or my team) have to run through meta/external-id fixes at least once
a week because of this.

[0]
https://groups.google.com/forum/#!msg/repo-discuss/tZ1tYQwbeLY/xSZhIQ20EQAJ

On 9/24/19 8:45 AM, Luca Milanesio wrote:
>
>
>> On 24 Sep 2019, at 16:36, Michael Croes <myc...@gmail.com

>> <mailto:myc...@gmail.com>> wrote:
>>
>> Hi Luca,
>>
>> The externalIds were unique, but for every account I had
>> gerrit:<username> and mailto:<email> externalIds. Obviously the mailto
>> externalId had an email set, and for some accounts there also was an
>> email set on the gerrit:<user> externalId.
>
> Ah, I do understand now.
>
> I believe we should invest more in the "init step" and detect those
> situation during upgrade.
> It is very unpleasant finding out *after* the upgrade that people cannot
> login anymore.
>
> I do understand and share your pain.
> I am preparing a talk at the next forthcoming Gerrit User Summit in

> Sunnyvale (gus2019.eventbrite.com <http://gus2019.eventbrite.com>) and I

> will definitely mention the "consistency check" step.
> I'll try also to write some tools to help people migrating and detect
> those issues early on.
>
>> I actually couldn't even push refs/meta/external-ids as it was in
>> Gerrit due to the duplicate email values.
>
> Yes, you should sort out those issues on the server side, by pushing
> directly to the repo bypassing Gerrit in that case.
>>
>> So I have to disagree with you, nevertheless thanks for your response.
>
> It's good to disagree at times, as long as we keep talking :-)
> And thanks for your feedback, much appreciated.
>
> Luca.
>

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com
> <mailto:repo-discuss...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/repo-discuss/6CE50193-DB86-4F5F-BB52-22C4CD6655C7%40gmail.com
> <https://groups.google.com/d/msgid/repo-discuss/6CE50193-DB86-4F5F-BB52-22C4CD6655C7%40gmail.com?utm_medium=email&utm_source=footer>.

[Karl]

I (or my team) have to run through meta/external-id fixes at least once
a week because of this.

I and my staffs are in the same boat after upgrading to 2.16.x.

It's very annoying :-(

[Luca Milanesio]

Hi Karl,

is your problem the same as https://bugs.chromium.org/p/gerrit/issues/detail?id=11246 ?

Luca.

[Luca Milanesio]

BTW, I've escalated the issue to a P1, it's pretty serious IMHO.

Going to see if I can reproduce it and create a test case for it.

We clearly missed the test-case in our tests and that's why it broke. So, we need to create a failing test first and fix the problem afterwards.

Luca.

[Karl]

Hi Karl,

is your problem the same as https://bugs.chromium.org/p/gerrit/issues/detail?id=11246 ?

Partly yes. 

However, in our case, newly created user also cannot login due to:

WARN  com.google.gerrit.server.account.AccountManager : Email user.registered.after.gerrit.upd...@example.net is already assigned to account GERRIT_USER_ID; cannot create external ID gerrit:sAMAccountName-for-this-user with the same email for account GERRIT_USER_ID_GERRIT_SOMEHOW_TRIED_TO_CREATE.

I suspect that the following issue is also related in our case:

https://bugs.chromium.org/p/gerrit/issues/detail?id=11498

Because most of our userid are numeric digits only (7 digits). 

Related discussions:

https://groups.google.com/forum/#!topic/repo-discuss/wCt_ePUoKg4

https://groups.google.com/forum/#!topic/repo-discuss/tZ1tYQwbeLY

https://groups.google.com/forum/#!topic/repo-discuss/sdf67teSEo8

[Luca Milanesio]

On 26 Sep 2019, at 02:25, Karl <karukomi...@jp.panasonic.com> wrote:

Hi Karl,

is your problem the same as https://bugs.chromium.org/p/gerrit/issues/detail?id=11246 ?

Partly yes. 

However, in our case, newly created user also cannot login due to:

WARN  com.google.gerrit.server.account.AccountManager : Email user.registered.after.gerrit.upd...@example.net is already assigned to account GERRIT_USER_ID; cannot create external ID gerrit:sAMAccountName-for-this-user with the same email for account GERRIT_USER_ID_GERRIT_SOMEHOW_TRIED_TO_CREATE.

I suspect that the following issue is also related in our case:

https://bugs.chromium.org/p/gerrit/issues/detail?id=11498

Because most of our userid are numeric digits only (7 digits). 

Oh, that's a completely different issue, and very tricky indeed.

I started fixing one use-case (the simpleusergroup plugin), but there are for sure *many many others*.

I proposed, also, to explicitly stating that the use-case isn't supported ... however, many people said that they are actually using it :-(

Luca.

Related discussions:

https://groups.google.com/forum/#!topic/repo-discuss/wCt_ePUoKg4

https://groups.google.com/forum/#!topic/repo-discuss/tZ1tYQwbeLY

https://groups.google.com/forum/#!topic/repo-discuss/sdf67teSEo8

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/89bd4e28-b82a-4095-976e-93ef76934c9d%40googlegroups.com.

[lucamilanesio]

Please look carefully at the additional test-cases that I have added to the AccountManagerIT:

https://gerrit-review.googlesource.com/c/gerrit/+/238833/6/javatests/com/google/gerrit/acceptance/api/accounts/AccountManagerIT.java#442

I have included exactly the use-cases that you guys have described and, no surprise, they were failing before my fix.

The failure was exactly the one you mentioned.

Therefore, if the change https://gerrit-review.googlesource.com/c/gerrit/+/238833 gets successfully reviewed and merged, you guys can avoid using the "external-ids" cleanup by hand, and the users would just be able again to login via LDAP without issues.

P.S. As soon as the build is done on the CI, you can download the release.war and try by yourself.

HTH

Luca.

> To unsubscribe, email repo-d...@googlegroups.com

> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to repo-d...@googlegroups.com
> <mailto:repo-discuss+unsub...@googlegroups.com>.

[lucamilanesio]

On Friday, September 27, 2019 at 12:45:25 AM UTC+1, lucamilanesio wrote:

Please look carefully at the additional test-cases that I have added to the AccountManagerIT:

https://gerrit-review.googlesource.com/c/gerrit/+/238833/6/javatests/com/google/gerrit/acceptance/api/accounts/AccountManagerIT.java#442

I have included exactly the use-cases that you guys have described and, no surprise, they were failing before my fix.

The failure was exactly the one you mentioned.

Therefore, if the change https://gerrit-review.googlesource.com/c/gerrit/+/238833 gets successfully reviewed and merged, you guys can avoid using the "external-ids" cleanup by hand, and the users would just be able again to login via LDAP without issues.

P.S. As soon as the build is done on the CI, you can download the release.war and try by yourself.

An additional element to the story, I have found also the root cause of the problem:

https://gerrit-review.googlesource.com/c/gerrit/+/169970

The above change introduced a stricter check for avoiding different accounts to have the same e-mail ... however, the check was missing the account id and only checked that "the e-mail does not exist anywhere in Gerrit", which is incorrect. You could have a secondary e-mail (mailto: record) that gets promoted to preferred e-mail (the 'mail' field in the 'gerrit:' record) and that *IS SUPPORTED* scenario. Only different accounts with the same e-mail *IS NOT VALID* and my fix actually improves the check by pulling in the account-id also.

Waiting for Edwin to double-check and hopefully approve the fix.

HTH

Luca.

[Kévin Raymond]

> I am preparing a talk at the next forthcoming Gerrit User Summit in Sunnyvale (gus2019.eventbrite.com) and I will definitely mention the "consistency check" step.
> I'll try also to write some tools to help people migrating and detect those issues early on.

That would be awesome.
I have actually and old bug in my process with Gerrit 2.14 where
Gerrit user changing email address leads to the "gerrit:<username>"
entry deletion.
I shall have open a ticket for that by that time but I still use the
workaround regularly
https://groups.google.com/forum/#!msg/repo-discuss/R4J5Qmyfymg/MN6cq0ZTFAAJ

Planning to upgrade, we would probably face similar issues.

[Anna Freiholtz]

My problem started as with comment 5 in the linked bug. When my user tries to login I see this warning:

[HTTP-41357] WARN  com.google.gerrit.server.account.AccountManager : Email o...@test.com is already assigned to account 204; cannot create external ID gerrit:one with the same email for account 217.

And it seems like every time he tried to login, Gerrit has created a new account.

I removed his mail address so the user had no mail addresses:

[HTTP-42742] ERROR com.google.gerrit.httpd.auth.container.HttpLoginServlet : Unable to authenticate user "one"

com.google.gerrit.server.account.AccountException: Cannot assign external ID "username:one" to account 228; external ID already in use.

Using Gerrit 2.16.10

Do you think the fix you mentioned below will help me get my user going again? How should I proceed with all the incorrect accounts?

BR,

Anna

Den onsdag 25 september 2019 kl. 11:03:41 UTC+2 skrev lucamilanesio: