New Versions Of Plugins Don't Run But Are Clearly Installed W/O Error

[Dave Castagna (Motorola Mobility)]

So I'm sure someone here knows what's going on here.  There is a missing piece to the puzzle that I just can't find.

Our team has developed several plugins for Gerrit.  (We are using 2.8.4 and are trying to upgrade to 2.10.2 but we need to get our plugins squared away first.)

We are trying to work out the correct process for updating our plugin code, building a new jar and installing the jar on the server.  But we're having major difficulties getting Gerrit to actually run new versions of whatever plugin we update.

Here's a typical scenario:

1. I have a plugin, let's say it's version 2.0.  Let's call it "george".  It is installed and running fine in Gerrit.  I have a little bit of code in this plugin that writes "2.0" out to a plugin-specific file.
   
2. In my plugin code I make a change.  Nothing changes structurally (the way the code is organized).  No new classes are created and no new dependencies are added.
   
3. I also change the "2.0" output to be "2.1".
   
4. I recompile the plugin JAR file "george-2.1.jar".
   
5. I remove the plugin (gerrit plugin rm george).
   
6. I install the new version (gerrit plugin install ....../george-2.1.jar).  No errors in error_log.  No errors on the command line.  I wait for the plugin cleanup messages in the error_log file.
   
7. I watch my plugin's log file and I see it print out "2.0" and not the expected "2.1".
   
8. I check the checksum of the "george-2.1.jar" file that is sitting in $GERRIT_HOME/plugins and it is the new file.
   
9. I even unjar the file and reverse compile the class file with the "2.0"/"2.1" in it and it clearly shows "2.1".
   

So that's weird.

So I try this:

1. I remove the plugin again.
   
2. I remove the "george*.disabled" from $GERRIT_HOME/plugins entirely.
   
3. I shut down Gerrit.
   
4. I remove the $GERRIT_HOME/tmp folder entirely as well.
   
5. I restart Gerrit.
   
6. I install the new version fresh (gerrit plugin install ....../george-2.1.jar).
   
7. I watch my plugin's log file and I see it print out "2.0" and not the expected "2.1".
   
8. I check the checksum of the "george.jar" file that is sitting in $GERRIT_HOME/plugins and it is the new file.
   
9. I even unjar the file and reverse compile the class file with the "2.0" and "2.1" in it and it clearly shows "2.1".
   

Okay I am now at a complete loss.

Why is Gerrit silently and smoothly taking the install but clearly running old code?

Where is Gerrit getting that old code?  What other caches exist?

I've done other experiments too and it is clear that there is some kind of cache somewhere in the system that is holding on to old plugin Jars.

But I can't for the life of me find it.

On a hunch I try this:

1. I go into my Java project for the "george" plugin and I refactor the package(s) so that the package names are different.
   
2. I recompile the plugin JAR file "george-2.1.jar".
   
3. I remove the plugin (gerrit plugin rm george).
   
4. I install the new version (gerrit plugin install ....../george-2.1.jar).  No errors in error_log.  No errors on the command line.  I wait for the plugin cleanup messages in the error_log file.
   
5. I watch my plugin's log file and I see it print out "2.1"!
   

Can anyone help me out here?

What is going on here?

Clearly I should not have to refactor my plugin's code on every upload.

Is there a decent guide to the dependency management and caching going on with Gerrit's plugin mechanism?

Note the problem gets even more complicated and much worse when my plugin incorporates libraries from other Java projects.

[David Ostrovsky]

Am Freitag, 10. Juli 2015 21:20:08 UTC+2 schrieb Dave Castagna (Motorola Mobility):

So I'm sure someone here knows what's going on here.  There is a missing piece to the puzzle that I just can't find.

Our team has developed several plugins for Gerrit.  (We are using 2.8.4 and are trying to upgrade to 2.10.2 but we need to get our plugins squared away first.)

We are trying to work out the correct process for updating our plugin code, building a new jar and installing the jar on the server.  But we're having major difficulties getting Gerrit to actually run new versions of whatever plugin we update.

Here's a typical scenario:

1. I have a plugin, let's say it's version 2.0.  Let's call it "george".  It is installed and running fine in Gerrit.  I have a little bit of code in this plugin that writes "2.0" out to a plugin-specific file.
   
2. In my plugin code I make a change.  Nothing changes structurally (the way the code is organized).  No new classes are created and no new dependencies are added.
   
3. I also change the "2.0" output to be "2.1".

I would stop here, pull cookbook-plugin, push my changes for review

and add exact steps what I was doing, my expectation and the failure

in the commit message. Hopefully, someone would look into it. The last

step is not that obvious in post-Shawn era for this project as this change

has shown: [1].

* [1] https://gerrit-review.googlesource.com/68795

[Björn Pedersen]

Hi,

I am not sure gerrit really supports dynamic reloading of plugins. So to upgrade a plugin you probably have to restart gerrit after installation of the new plugin.
What you observe after the refactoring is, that you are now using the new jar, insead of the old ones, that  where still loaded.

Björn

[Luca Milanesio]

Hi Bjorn,

Gerrit does support plugins reloading BUT it really depends on:

a) The plugin to declare himself to be reloadable

b) The plugin to behave correctly after reload

As the reload is based on the creation of a new plugin context on a different class-loader, it relies on the plugin “Good will” of de-allocating the resource upload reload.

HTH

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Castagna (Motorola Mobility)]

Thanks Luca.  I think that is closer to my actual problem.  Do you have an example of what resources need to be "de-allocated"?

Also regarding:

a) The plugin to declare himself to be reloadable

The plugin is declared like this: 

<Gerrit-ReloadMode>restart</Gerrit-ReloadMode>

Are you suggesting I change that to "reload" and then replacing the plugin with a "gerrit plugin rm" -> "gerrit plugin install" will work?  I assuming you are referring to this verbage in the documentation?:

If a plugin holds an exclusive resource that must be released before loading the plugin again (for example listening on a network port or acquiring a file lock) the manifest must declare Gerrit-ReloadMode to be restart. Otherwise the preferred method of reload will be used, as it enables the server to hot-patch an updated plugin with no down time.

I will give this a try.

BUT:

Why would any of this apply after a restart of Gerrit?  What is being cached and where?  Because when I restart Gerrit the old plugin behavior is what is observed.  Even after deleting the $GERRIT_HOME/tmp folder between shutdown and startup of Gerrit itself the old plugin behavior is still seen.

[Luca Milanesio]

Hi Dave,

see my feedback below.

On 13 Jul 2015, at 17:33, Dave Castagna (Motorola Mobility) <cast...@motorola.com> wrote:

Thanks Luca.  I think that is closer to my actual problem.  Do you have an example of what resources need to be "de-allocated"?

Also regarding:

a) The plugin to declare himself to be reloadable

The plugin is declared like this: 

<Gerrit-ReloadMode>restart</Gerrit-ReloadMode>

Are you suggesting I change that to "reload" and then replacing the plugin with a "gerrit plugin rm" -> "gerrit plugin install" will work?  I assuming you are referring to this verbage in the documentation?:

If a plugin holds an exclusive resource that must be released before loading the plugin again (for example listening on a network port or acquiring a file lock) the manifest must declare Gerrit-ReloadMode to be restart. Otherwise the preferred method of reload will be used, as it enables the server to hot-patch an updated plugin with no down time.

Restart => you need to restart Gerrit and plugin cannot be reloaded dynamically.

IF your plugin is holding resources, it can still be reloadable if your plugin is going to implement the com.google.gerrit.extensions.events.LifecycleListener interface and close and re-allocate the necessary resources on the start()/stop() methods.

I will give this a try.

BUT:

Why would any of this apply after a restart of Gerrit?  What is being cached and where?  Because when I restart Gerrit the old plugin behavior is what is observed.  Even after deleting the $GERRIT_HOME/tmp folder between shutdown and startup of Gerrit itself the old plugin behavior is still seen.

This is not possible: if you clean-up the old plugin from $GERRIT_SITE/plugins and restart Gerrit should be more than enough.

Luca.

[David Castagna]

Okay thanks Luca - I think that's enough to go on for now.

But to be clear - I would not have logged this post if the plugin reloading/restarting was behaving like I expected it to.

I very clearly and obviously have seen Gerrit continue to execute behavior from an older version of a plugin even after wiping out folders in the gerrit installation and restarting.

It is possible I did not wipe out both the $GERRIT_HOME/plugins AND the $GERRIT_HOME/tmp folders during my tests.

If I can reproduce the behavior reliably I will report back.

--

Make the thing you want to see exist in the world.   -Chris Hardwick

Be a maker, Be a doer, Be a creator.  Have ideas and do them.    -Alexis Ohanian

[Luca Milanesio]

Yes, please report the steps to reproduce the problem if that happens again.

Luca

Sent from my iPhone