Gerrit HA plugin issue

[Oleksandr Mykytenko]

I'm using Gerrit 2.13.5 and there are 2 instances in Multi-Master system. I'm using gerrit high-availability plugin for syncing caches, events and indexes. Everything works fine exclude one strange thing: when I create new user, I can login only on one Gerrit server, in another one I receive the following error: [2017-08-04 09:38:31,391] [HTTP-57] ERROR com.google.gerrit.httpd.auth.container.HttpLoginServlet : Unable to authenticate user "user_test" com.google.gerrit.server.account.AccountException: Cannot assign external ID "gerrit:user_test" to account 1000256; external ID already in use. Smth like HA plugin doesn't synchronize users in indexes on both Gerrit. Please note that all other actions (changes in Change requests, in projects) syncs fine.

[Hugo Arès]

Just some basic validation, are both masters sharing the same database and git repositories?

[Oleksandr Mykytenko]

Yes, both Gerrit use the same DB and directory with git repos

пʼятниця, 4 серпня 2017 р. 22:35:23 UTC+3 користувач Hugo Arès написав:

[Edwin Kempin]

Yes, you are getting this error because the the account index on that node is stale.

[1] would fix the login for you, but it's only contained since 2.14.2. Anyway if the account index is stale you would likely run into other issues as well.

[1] https://gerrit-review.googlesource.com/111098

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oleksandr Mykytenko]

Thank you, do you mean new version of Gerrit app or new version of HA plugin? When will appear new version?

Maybe you can advice me some workarounds ?

понеділок, 7 серпня 2017 р. 09:26:14 UTC+3 користувач Edwin Kempin написав:

Yes, you are getting this error because the the account index on that node is stale.

[1] would fix the login for you, but it's only contained since 2.14.2. Anyway if the account index is stale you would likely run into other issues as well.

[1] https://gerrit-review.googlesource.com/111098

On Mon, Aug 7, 2017 at 8:03 AM, Oleksandr Mykytenko <alexandr....@gmail.com> wrote:

Yes, both Gerrit use the same DB and directory with git repos

пʼятниця, 4 серпня 2017 р. 22:35:23 UTC+3 користувач Hugo Arès написав:

Just some basic validation, are both masters sharing the same database and git repositories?

On Friday, August 4, 2017 at 7:45:35 AM UTC-4, Oleksandr Mykytenko wrote:

I'm using Gerrit 2.13.5 and there are 2 instances in Multi-Master system. I'm using gerrit high-availability plugin for syncing caches, events and indexes. Everything works fine exclude one strange thing: when I create new user, I can login only on one Gerrit server, in another one I receive the following error: [2017-08-04 09:38:31,391] [HTTP-57] ERROR com.google.gerrit.httpd.auth.container.HttpLoginServlet : Unable to authenticate user "user_test" com.google.gerrit.server.account.AccountException: Cannot assign external ID "gerrit:user_test" to account 1000256; external ID already in use. Smth like HA plugin doesn't synchronize users in indexes on both Gerrit. Please note that all other actions (changes in Change requests, in projects) syncs fine.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Edwin Kempin]

On Mon, Aug 7, 2017 at 8:33 AM, Oleksandr Mykytenko <alexandr....@gmail.com> wrote:

Thank you, do you mean new version of Gerrit app or new version of HA plugin? When will appear new version?

2.14.2 is a new Gerrit version and it's already available.

 

Maybe you can advice me some workarounds ?

Likely the HA plugin needs to be adapted to take care about replicating the account index.

 

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Oleksandr Mykytenko]

Can you tell me please who can adopt plugin and add this feature? Because I can't find any contacts for gerrit high-availability plugin.

Cheers, Oleksandr 

пʼятниця, 4 серпня 2017 р. 14:45:35 UTC+3 користувач Oleksandr Mykytenko написав:

[Edwin Kempin]

On Mon, Aug 7, 2017 at 8:44 AM, Oleksandr Mykytenko <alexandr....@gmail.com> wrote:

Can you tell me please who can adopt plugin and add this feature?

It's open source, so anyone can propose changes to the plugin.

 

Because I can't find any contacts for gerrit high-availability plugin.

 

Here you can find the people that worked on this plugin in the past:

  https://gerrit-review.googlesource.com/#/q/status:merged+project:plugins/high-availability

Hugo and Hector are the project owners.

 

Cheers, Oleksandr 

пʼятниця, 4 серпня 2017 р. 14:45:35 UTC+3 користувач Oleksandr Mykytenko написав:

I'm using Gerrit 2.13.5 and there are 2 instances in Multi-Master system. I'm using gerrit high-availability plugin for syncing caches, events and indexes. Everything works fine exclude one strange thing: when I create new user, I can login only on one Gerrit server, in another one I receive the following error: [2017-08-04 09:38:31,391] [HTTP-57] ERROR com.google.gerrit.httpd.auth.container.HttpLoginServlet : Unable to authenticate user "user_test" com.google.gerrit.server.account.AccountException: Cannot assign external ID "gerrit:user_test" to account 1000256; external ID already in use. Smth like HA plugin doesn't synchronize users in indexes on both Gerrit. Please note that all other actions (changes in Change requests, in projects) syncs fine.

--

[Oleksandr Mykytenko]

Hello, as I know you are the project owner for gerrit HA plugin. Can someone please add account index replication to HA plugin? I hope that this feature will fix my issue. Please see all messaging history above.

Thank you in advance.

пʼятниця, 4 серпня 2017 р. 22:35:23 UTC+3 користувач Hugo Arès написав:

Just some basic validation, are both masters sharing the same database and git repositories?

[Hector Oswaldo Caballero]

Hi Oleksandr,

This is in our TODO list but currently we are a bit busy preparing our upgrade to 2.14. We'll keep you posted when this is ready. 

BR,

Hector

[Luca Milanesio]

Hi Oleksandr,

bear in mind that creating accounts isn't a very frequent event: how many new accounts are created every day? every hour? every second?

I would recommend you to put in place a very simple script to workaround the problem:

- select min(account_id) as mi, max(account_id) as ma from accounts where registered_on > T

- for (( i=mi; i<ma; i++)); do curl -X POST http://gerrit.myserver.com/a/accounts/$i/index; done

If you put the two statements in an organised script and execute on a regular basis (e.g. every 1h or 30' or even less), you'll easily solve your problem in an effective way.

HTH

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[David Pursehouse]

Doesn't the plugin already support forwarding of account index?

[Oleksandr Mykytenko]

Thank you, it works perfect!

вівторок, 8 серпня 2017 р. 09:50:34 UTC+3 користувач lucamilanesio написав:

[Oleksandr Mykytenko]

I don't know. I'm not a developer. I'm just using this plugin and issue was appear.

вівторок, 8 серпня 2017 р. 13:27:44 UTC+3 користувач David Pursehouse написав:

[Luca Milanesio]

Apparently not, as Hector mentioned that is in their TODO list :-(

Luca.

[David Pursehouse]

It should be implemented since [1] unless we're talking about different things?

[1] https://gerrit-review.googlesource.com/#/c/plugins/high-availability/+/105081/

[Luca Milanesio]

Right, that definitely seems the one.

The change on high-availability plugin was on the stable-2.13 branch, possibly just the plugin needs to be upgraded to the latest 2.13?

@Hector: is then supposed to work as-is today? To which piece of work were you referring to?

Luca.

[Oleksandr Mykytenko]

I've built plugin from stable-2.13? but after restarting with new version of HA plugin Gerrit got new error:

[2017-08-09 10:45:06,347] [main] WARN  com.google.gerrit.server.plugins.PluginLoader : Cannot load plugin high-availability

java.lang.NoClassDefFoundError: com/google/gerrit/extensions/events/AccountIndexedListener

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at com.ericsson.gerrit.plugins.highavailability.index.IndexModule.configure(IndexModule.java:29)

at com.google.inject.AbstractModule.configure(AbstractModule.java:62)

at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)

at com.google.inject.AbstractModule.install(AbstractModule.java:122)

at com.ericsson.gerrit.plugins.highavailability.Module.configure(Module.java:51)

at com.google.inject.AbstractModule.configure(AbstractModule.java:62)

at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)

at com.google.inject.spi.Elements.getElements(Elements.java:110)

at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)

at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)

at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:232)

at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:236)

at com.google.gerrit.server.plugins.ServerPlugin.startPlugin(ServerPlugin.java:173)

at com.google.gerrit.server.plugins.ServerPlugin.start(ServerPlugin.java:155)

at com.google.gerrit.server.plugins.PluginLoader.runPlugin(PluginLoader.java:508)

at com.google.gerrit.server.plugins.PluginLoader.rescan(PluginLoader.java:435)

at com.google.gerrit.server.plugins.PluginLoader.start(PluginLoader.java:339)

at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:89)

at com.google.gerrit.pgm.Daemon.start(Daemon.java:311)

at com.google.gerrit.pgm.Daemon.run(Daemon.java:212)

at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:64)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:161)

at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:102)

at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:59)

at Main.main(Main.java:25)

Caused by: java.lang.ClassNotFoundException: com.google.gerrit.extensions.events.AccountIndexedListener

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 40 more

Version of Gerrit: 2.13.5

вівторок, 8 серпня 2017 р. 19:37:30 UTC+3 користувач David Pursehouse написав:

[Edwin Kempin]

The AccountIndexedListener extention point was added by [1] and is only available since 2.13.8.

Means you must upgrade your Gerrit server at least to 2.13.8 if you want to use this version of the HA plugin.

[1] https://gerrit-review.googlesource.com/104370

 

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[luca.mi...@gmail.com]

It is a bit unusual we changed the API in a patch release :-(

Is that a normal policy?

Luca

Sent from my iPhone

[Edwin Kempin]

On Wed, Aug 9, 2017 at 10:09 AM, <luca.mi...@gmail.com> wrote:

It is a bit unusual we changed the API in a patch release :-(

It only added some new API and existing API wasn't changed.

 

Is that a normal policy?

This was discussed at the last hackathon. IIRC we agreed to it so that Ericsson has a way to make HA work with Gerrit 2.13, otherwise they wouldn't be able to use Gerrit 2.13.

[Luca Milanesio]

On 9 Aug 2017, at 09:14, Edwin Kempin <eke...@google.com> wrote:

On Wed, Aug 9, 2017 at 10:09 AM, <luca.mi...@gmail.com> wrote:

It is a bit unusual we changed the API in a patch release :-(

It only added some new API and existing API wasn't changed.

Yes, but that means that we should in theory make sure that we build a plugin with the latest version of the stable branch, otherwise we may miss the new API introduced there.

 

Is that a normal policy?

This was discussed at the last hackathon. IIRC we agreed to it so that Ericsson has a way to make HA work with Gerrit 2.13, otherwise they wouldn't be able to use Gerrit 2.13.

In Ericsson they use HA for active failover, not workload balancing. The suggested workaround would have worked nicely (simple crontab to reindex newly created accounts) for a failover scenario.

Shall we document this policy in our Documentation set? Let me check and propose a patch ....

I am fine with either way, we just need to be sure that our policies are properly documented and shared with the Community, including the one outside the Hackathons ;-)

Luca.

[Luca Milanesio]

I've documented the policy change at [1]

It was  previously mentioning the dependency on only major.minor ... and not the fact that a plugin may depend on a specific patch-level API.

I've amended it to reflect the current situation: let me know if that matches what has been done and agreed :-)

Luca.

[1] https://gerrit-review.googlesource.com/c/118830