Issues with LDAP account logging in 2.16.8

84 views
Skip to first unread message

Andrew Grimberg

unread,
Jun 17, 2019, 5:15:30 PM6/17/19
to Gerrit Code Review
We recently upgraded a Gerrit system from 2.14.?? up to 2.16.8. This
system is LDAP backed.

Since the upgrade we've got one user that had their email address change
in LDAP which was causing issues with their logging in.

In trying to fix their issues I did some "fixes" to their NoteDB
external-ids records and have their records in a state that looks
correct. The issue that we're running into is that they _still_ can't login.

I keep getting an error like:

[2019-06-17 10:44:11,332] [HTTP-938670] WARN
com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'REDACTED' failed
to sign in
com.google.gerrit.server.account.AccountException: Cannot assign
external ID "username:REDACTED" to account 344; external ID already in use.

The account has accountId of 308 and their records all are correct

echo -n 'username:REDACTED' | sha1sum => is the same for the record that
already is in meta/external-ids with accountId 308, but it keeps trying
to create a new one.

Additionally the 'gerrit:REDACTED' entry all matches up correctly too
and everything that I see in the users/08/308 branch seems correct as well.

Has anyone else run into something like this? If so, what was the fix?

-Andy-

signature.asc

David Pursehouse

unread,
Jun 17, 2019, 8:11:31 PM6/17/19
to Andrew Grimberg, Gerrit Code Review
Have you tried to run the consistency checker [1]?  This will tall you if there are still any issues with the accounts or external IDs.



 
-Andy-

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/6434465a-5824-c720-006e-dbb16a38840a%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Karl

unread,
Jun 17, 2019, 9:33:22 PM6/17/19
to Repo and Gerrit Discussion
Related issue:

As a workaround, we modify records in All-Users.git which belong to users cannot sign in.  

2019年6月18日火曜日 6時15分30秒 UTC+9 Andrew Grimberg:

Andrew Grimberg

unread,
Jun 18, 2019, 3:54:14 PM6/18/19
to Karl, Repo and Gerrit Discussion
I've pretty much tried all of this. The only thing I have yet to do is
remove the all the external ID records and their user record. I don't
really want to go losing all the reviews that they've got in the system.

-Andy-
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com
> <mailto:repo-discuss...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/repo-discuss/f7a84424-e306-46cb-b043-1281c6f79c74%40googlegroups.com
> <https://groups.google.com/d/msgid/repo-discuss/f7a84424-e306-46cb-b043-1281c6f79c74%40googlegroups.com?utm_medium=email&utm_source=footer>.
signature.asc

Andrew Grimberg

unread,
Jun 18, 2019, 3:56:03 PM6/18/19
to David Pursehouse, Gerrit Code Review
On 6/17/19 5:11 PM, David Pursehouse wrote:

--[snip]--

>
> Have you tried to run the consistency checker [1]?  This will tall you
> if there are still any issues with the accounts or external IDs.
>
> http://gerrit-documentation.storage.googleapis.com/Documentation/2.16/rest-api-config.html#check-consistency

Ah! I didn't know about this (answers a question I had in another thread
a few weeks ago). Having run that against this particular Gerrit
instance, I do have some issues on accounts (not the problem account),
but nothing in the external_ids check :-/

-Andy-

signature.asc

Sven Selberg

unread,
Jun 19, 2019, 11:13:15 AM6/19/19
to Repo and Gerrit Discussion
We encountered this and it was that the username in one of the external ids had wrong case, probably something that was not fully corrected during ReviewDb and then migrated to NoteDb

All-Users(refs/meta/external-ids)
cat <one file>
[externalId "username:ottos"] 

cat <other file>
[externalId "gerrit:Ottos"]

I don't know how to set this straight in NoteDb...

Tracked by:

/Sven

Andrew Grimberg

unread,
Jun 19, 2019, 12:28:22 PM6/19/19
to Sven Selberg, Repo and Gerrit Discussion
On 6/19/19 8:13 AM, Sven Selberg wrote:
> We encountered this and it was that the username in one of the external
> ids had wrong case, probably something that was not fully corrected
> during ReviewDb and then migrated to NoteDb
>
> All-Users(refs/meta/external-ids)
> cat <one file>
> [externalId "username:ottos"] 
>
> cat <other file>
> [externalId "gerrit:Ottos"]
>
> I don't know how to set this straight in NoteDb...
>
> Tracked by:
> https://bugs.chromium.org/p/gerrit/issues/detail?id=10910
>
> /Sven

Not the problem this time. I _think_ I've tracked it down now, but I'm
not 100% certain as my customer hasn't responded to my most recent request.

If so, it's tied directly to the fact that we have an open identity
system that allows folks to create accounts easily. We have a hard
requirement on one uniq email address per account, but folks can easily
change that. Which means that they can change their email address that
Gerrit ends up seeing. This particular customer changed the address on
one account to something else (which I've now fixed their
meta/external-ids entries and user node to match) and created a new
account in our identity system with the old email address.

Hopefully now that I've made my most recent fixes they'll be able to login.

The next thing that I am now trying to puzzle out is how to clean up
some of the user node consistency issues that I'm seeing. I've got an
account that has no meta/externa-ids defined but has a user branch. I
want completely axe the account from the system as there is no uid
associated with it like there is supposed to be as well, but I'm unsure
what would happen if I just delete the user branch.

-Andy-

> On Tuesday, June 18, 2019 at 9:56:03 PM UTC+2, Andrew Grimberg wrote:
>
> On 6/17/19 5:11 PM, David Pursehouse wrote:
>
> --[snip]--
>
> >
> > Have you tried to run the consistency checker [1]?  This will tall
> you
> > if there are still any issues with the accounts or external IDs.
> >
> >
> http://gerrit-documentation.storage.googleapis.com/Documentation/2.16/rest-api-config.html#check-consistency
> <http://gerrit-documentation.storage.googleapis.com/Documentation/2.16/rest-api-config.html#check-consistency>
>
>
> Ah! I didn't know about this (answers a question I had in another
> thread
> a few weeks ago). Having run that against this particular Gerrit
> instance, I do have some issues on accounts (not the problem account),
> but nothing in the external_ids check :-/
>
> -Andy-
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com
> <mailto:repo-discuss...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/repo-discuss/6a8ae326-fdd9-4e57-80d7-1609667380f2%40googlegroups.com
> <https://groups.google.com/d/msgid/repo-discuss/6a8ae326-fdd9-4e57-80d7-1609667380f2%40googlegroups.com?utm_medium=email&utm_source=footer>.
signature.asc

Sven Selberg

unread,
Jun 20, 2019, 4:04:34 AM6/20/19
to Repo and Gerrit Discussion
Could anyone guide me on how to manually correct a faulty external-id in All-Users(refs/meta/external-ids)?
How is the file-name of the external-id constructed?

/Sven

David Pursehouse

unread,
Jun 20, 2019, 5:01:51 AM6/20/19
to Sven Selberg, Repo and Gerrit Discussion
On Thu, Jun 20, 2019 at 5:04 PM Sven Selberg <sven.s...@axis.com> wrote:
Could anyone guide me on how to manually correct a faulty external-id in All-Users(refs/meta/external-ids)?
How is the file-name of the external-id constructed?

It's the sha1 of the external-id.  For example given the external-id "username:foobar":

echo -n "username:foobar" | openssl sha1
(stdin)= dcd802a3dca12e4649cababa02247510a1e4e181

then the file-name for that external-id is dcd802a3dca12e4649cababa02247510a1e4e181


 

/Sven

On Wednesday, June 19, 2019 at 5:13:15 PM UTC+2, Sven Selberg wrote:
We encountered this and it was that the username in one of the external ids had wrong case, probably something that was not fully corrected during ReviewDb and then migrated to NoteDb

All-Users(refs/meta/external-ids)
cat <one file>
[externalId "username:ottos"] 

cat <other file>
[externalId "gerrit:Ottos"]

I don't know how to set this straight in NoteDb...

Tracked by:

/Sven

On Tuesday, June 18, 2019 at 9:56:03 PM UTC+2, Andrew Grimberg wrote:
On 6/17/19 5:11 PM, David Pursehouse wrote:

--[snip]--

>
> Have you tried to run the consistency checker [1]?  This will tall you
> if there are still any issues with the accounts or external IDs.
>
> http://gerrit-documentation.storage.googleapis.com/Documentation/2.16/rest-api-config.html#check-consistency

Ah! I didn't know about this (answers a question I had in another thread
a few weeks ago). Having run that against this particular Gerrit
instance, I do have some issues on accounts (not the problem account),
but nothing in the external_ids check :-/

-Andy-

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Sven Selberg

unread,
Jun 20, 2019, 5:16:32 AM6/20/19
to Repo and Gerrit Discussion


On Thursday, June 20, 2019 at 11:01:51 AM UTC+2, David Pursehouse wrote:
On Thu, Jun 20, 2019 at 5:04 PM Sven Selberg <sven....@axis.com> wrote:
Could anyone guide me on how to manually correct a faulty external-id in All-Users(refs/meta/external-ids)?
How is the file-name of the external-id constructed?

It's the sha1 of the external-id.  For example given the external-id "username:foobar":

echo -n "username:foobar" | openssl sha1
(stdin)= dcd802a3dca12e4649cababa02247510a1e4e181

then the file-name for that external-id is dcd802a3dca12e4649cababa02247510a1e4e181


Thanks, that's it!
I tried SHA1 but I must have fumbled somehow cause I didn't get the correct result before.

For clarity the file-name is actually:

    dc/d802a3dca12e4649cababa02247510a1e4e181
 

 
To unsubscribe, email repo-d...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

Andrew Grimberg

unread,
Jun 20, 2019, 2:23:55 PM6/20/19
to Sven Selberg, Repo and Gerrit Discussion
On 6/20/19 2:16 AM, Sven Selberg wrote:
> On Thursday, June 20, 2019 at 11:01:51 AM UTC+2, David Pursehouse wrote:
>
> On Thu, Jun 20, 2019 at 5:04 PM Sven Selberg <sven....@axis.com
> <javascript:>> wrote:
>
> Could anyone guide me on how to manually correct a faulty
> external-id in All-Users(refs/meta/external-ids)?
> How is the file-name of the external-id constructed?
>
>
> It's the sha1 of the external-id.  For example given the external-id
> "username:foobar":
>
> echo -n "username:foobar" | openssl sha1
> (stdin)= dcd802a3dca12e4649cababa02247510a1e4e181
>
> then the file-name for that external-id is
> dcd802a3dca12e4649cababa02247510a1e4e181
>
>
> Thanks, that's it!
> I tried SHA1 but I must have fumbled somehow cause I didn't get the
> correct result before.
>
> For clarity the file-name is actually:
>
>     dc/d802a3dca12e4649cababa02247510a1e4e181

Depends upon how many external-ids you have in the system. I don't know
at what point all the ids get sharded but a fresh install of Gerrit will
have them all in a flat structure and then at some tipping point they
get sharded like that.

For all I know you might end up with sub-shards if a particular one gets
too full.

-Andy-

signature.asc

Tech Advantage

unread,
Jun 20, 2019, 3:02:46 PM6/20/19
to Repo and Gerrit Discussion
Reply all
Reply to author
Forward
0 new messages