Error during db migration when upgrading from 2.14.1 directly to 2.16.2

[Mike Lupo]

Hello all,

I'm trying to update my gerrit instance to 2.16.2 from 2.14.1.

I'm encountering an error during the db migration.

What do I do to rectify this?

Details below...

When I run:

java -jar gerrit.war init -d /gerrit

I go through the interview accepting all of the defaults.

The problem occurs when the initializer attempts to migrate the schema.

Upgrading schema to 151 ...

Upgrading schema to 152 ...

Upgrading schema to 153 ...

Upgrading schema to 154 ...

Upgrading schema to 155 ...

Upgrading schema to 156 ...

Upgrading schema to 157 ...

Upgrading schema to 158 ...

Upgrading schema to 159 ...

Upgrading schema to 160 ...

Upgrading schema to 161 ...

Upgrading schema to 162 ...

Upgrading schema to 163 ...

Upgrading schema to 164 ...

Upgrading schema to 165 ...

Upgrading schema to 166 ...

Upgrading schema to 167 ...

Upgrading schema to 168 ...

Upgrading schema to 169 ...

Upgrading schema to 170 ...

Migrating data to schema 151 ...

Exception in thread "main" com.google.gwtorm.server.OrmException: Cannot upgrade schema

at com.google.gerrit.server.schema.SchemaUpdater.update(SchemaUpdater.java:113)

at com.google.gerrit.pgm.init.BaseInit$SiteRun.upgradeSchema(BaseInit.java:383)

at com.google.gerrit.pgm.init.BaseInit.run(BaseInit.java:144)

at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:226)

at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:122)

at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:66)

at Main.main(Main.java:28)

Caused by: org.postgresql.util.PSQLException: ERROR: column "created_on" of relation "account_groups" does not exist

  Position: 27

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)

at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)

at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)

at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)

at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:120)

at com.google.gerrit.server.schema.Schema_151.migrateData(Schema_151.java:54)

at com.google.gerrit.server.schema.SchemaVersion.migrateData(SchemaVersion.java:155)

at com.google.gerrit.server.schema.SchemaVersion.upgradeFrom(SchemaVersion.java:94)

at com.google.gerrit.server.schema.SchemaVersion.check(SchemaVersion.java:85)

at com.google.gerrit.server.schema.SchemaUpdater.update(SchemaUpdater.java:111)

... 11 more

[David Ostrovsky]

Can you try to do the upgrade in 2 steps:

1. Upgrade to the latest 2.15.8 release.
2. Upgrade to 2.16.2.

[Mike Lupo]

Seems that there is no 2.15.8 release. Shall I try rc4? or .7?

git tag --list | grep 2.15.

v2.15-rc0

v2.15-rc1

v2.15-rc2

v2.15-rc3

v2.15-rc4

v2.15.1

v2.15.2

v2.15.3

v2.15.4

v2.15.5

v2.15.6

v2.15.7

[Luca Milanesio]

On 29 Dec 2018, at 20:08, Mike Lupo <mike...@gmail.com> wrote:

Seems that there is no 2.15.8 release. Shall I try rc4? or .7?

2.15.7

Luca.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Lupo]

Thanks Luca.

Build failed. Bazel isn't happy. Here are my steps. Bazel version is 0.21.0

git checkout v2.15.7  

M plugins/commit-message-length-validator

M plugins/download-commands

M plugins/hooks

M plugins/replication

M plugins/reviewnotes

M plugins/singleusergroup

Previous HEAD position was a3ab2e1... Release 2.15-rc4

HEAD is now at d93736f... Set version to 2.15.7

mlupo@prod-gerrit-app04 ~/gerrit ((HEAD detached at v2.15.7)) $ git submodule update --init 

Submodule path 'plugins/commit-message-length-validator': checked out '4f6b685e12e34a4f583cf84ba1c58ccc2b75e8b0'

Submodule path 'plugins/download-commands': checked out 'cfed9533017919eba8ae89e27929651c7d37c0a5'

Submodule path 'plugins/hooks': checked out '75df5d54fcb0b56a799ae1dc3aa2f88cbc8e3dc3'

Submodule path 'plugins/replication': checked out '0fe68ab58cccfae23d113a22d3ce66296b7e9fae'

Submodule path 'plugins/reviewnotes': checked out '52d66d97f7c87a18c50840ab453ca84026b90738'

Submodule path 'plugins/singleusergroup': checked out '0ffbdf1e5c3daa9d118ee9f53d5c00497cff6454'

mlupo@prod-gerrit-app04 ~/gerrit ((HEAD detached at v2.15.7)) $ bazel build gerrit

WARNING: detected http_proxy set in env, setting no_proxy for localhost.

Starting local Bazel server and connecting to it...

INFO: Invocation ID: 02d33bda-5f0b-4dea-b109-111e04996615

ERROR: /u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/com_google_protobuf/BUILD:597:1: Traceback (most recent call last):

File "/u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/com_google_protobuf/BUILD", line 597

internal_gen_well_known_protos_java(srcs = WELL_KNOWN_PROTOS)

File "/u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/com_google_protobuf/protobuf.bzl", line 266, in internal_gen_well_known_protos_java

Label(("%s//protobuf_java" % REPOSITOR...))

File "/u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/com_google_protobuf/protobuf.bzl", line 266, in Label

REPOSITORY_NAME

The value 'REPOSITORY_NAME' has been removed in favor of 'repository_name()', please use the latter (https://docs.bazel.build/versions/master/skylark/lib/native.html#repository_name). You can temporarily allow the old name by using --incompatible_package_name_is_a_function=false

ERROR: /u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/com_google_protobuf/BUILD:380:1: Target '@com_google_protobuf//:android' contains an error and its package is in error and referenced by '@com_google_protobuf//:protoc'

ERROR: /u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/com_google_protobuf/BUILD:380:1: Target '@com_google_protobuf//:windows' contains an error and its package is in error and referenced by '@com_google_protobuf//:protoc'

ERROR: /u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/com_google_protobuf/BUILD:380:1: Target '@com_google_protobuf//:windows_msvc' contains an error and its package is in error and referenced by '@com_google_protobuf//:protoc'

ERROR: /u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/io_bazel_rules_closure/java/io/bazel/rules/closure/webfiles/BUILD:38:1: every rule of type proto_library implicitly depends upon the target '@com_google_protobuf//:protoc', but this target could not be found because of: Target '@com_google_protobuf//:protoc' contains an error and its package is in error

ERROR: /u0/mlupo/.cache/bazel/_bazel_mlupo/cc4e6c550fdd580109bff31a4e5dadaf/external/io_bazel_rules_closure/java/io/bazel/rules/closure/webfiles/BUILD:17:1: Target '@com_google_protobuf//:protobuf_java' contains an error and its package is in error and referenced by '@io_bazel_rules_closure//java/io/bazel/rules/closure/webfiles:webfiles'

ERROR: Analysis of target '//:gerrit' failed; build aborted: Analysis failed

INFO: Elapsed time: 8.170s

INFO: 0 processes.

FAILED: Build did NOT complete successfully (207 packages loaded, 1861 targets configured)

    Fetching @polymer-resin; fetching 4s

    Fetching @iron-flex-layout; fetching 4s

    Fetching @font-roboto; fetching

    Fetching @iron-menu-behavior; fetching

    Fetching @ant; fetching

    Fetching @polymer; fetching

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote:

[Mike Lupo]

Answering my own dilemma with the bazel build. Seems my bazel version (0.21.0) being much newer than what gerrit 2.15.x was built with, I had to include the parameter, "--incompatible_package_name_is_a_function=false"

So in context:

 bazel build gerrit --incompatible_package_name_is_a_function=false

Build succeeded.

I'll perform the update and will advise on the outcome.

Thanks,

Mike

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote:

[Mike Lupo]

David, Luca

Thanks so much for your support over the weekend. This is when I'm able to do such things as our users are not needing the system.

The answer was indeed to first upgrade to 2.15.7 and then to 2.16.2.

If I have any other issues, I'll start a new thread.

Regards,

Mike

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote:

[David Ostrovsky]

Great, that it worked. Yes Bazel incompatibility is a known issue. Note that the option you passed to fix the problem was removed on Bazel master and will not be included any more in upcoming Bazel release (0.22).

You have a number of options:
1. Download pre-built binaries from the official Gerrit page or Maven Central.
2. That doesn't help if you have to build custom plugins. In that case: checkout the tag you want to build, e.g.: v2.15.7 and check WORKSPASE file for minimal supported Bazel version. Deinstall all other Bazel versions and install exactly the same Bazel version that was used to build that Gerrit release. This should always work. (And yes, we should and will find a better way to support that use case.)

[Mike Lupo]

Thanks David.
I got the site up and running but my SAML SSO plugin is encountering a runtime error. It would be great if gerrit supported SAML out of the box so we don’t have to rely on third-party plugins for something that is becoming a very popular means of logging in.

Anyway...for me it may be something as simple as reconfiguring my Apache front end. I’ll look more into it when I’m fresh tomorrow.

Thanks again,
Mike

Sent from my iPhone

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---

> You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/FPfy_Ng8eGQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.

[David Ostrovsky]

Have you noticed a series of PRr on gerrit-saml-plugin on GitHub that I uploaded recently? I upgraded pac4j and transitive dependencies and added Bazel build to this plugin. I also did basic tests against KeyKloack IdP and it worked. It would be great if you could apply the PRs and test and provide feedback.

[mike...@aol.com]

David, I didn't notice them till you posted this.

Happy to try out your commits, I immediately pulled them all together in one branch. I am encountering a build error.

Here were my steps:

1) pull all your commits together in logical order in one branch from all 4 PRs.

2) mv the file external_plugin_deps.bzl from the gerrit-saml-plugin to ../ and verify that we've overwritten the dummy file.

3) Run the comand, bazel build plugins/gerrit-saml-plugin

Build starts and I can see it pulling down the dependencies.

Build ultimately fails with: (hoping the code curly brackets do the right thing with this copy/paste. wishing we had a "preview" window)

 /u0/mlupo/gerrit/plugins/gerrit-saml-plugin/BUILD:3:1: Building plugins/gerrit-saml-plugin/libgerrit-saml-plugin__plugin-class.jar (3 source files) and running annotation processors (AutoAnnotationProcessor, AutoValueProcessor) failed (Exit 1)

plugins/gerrit-saml-plugin/src/main/java/com/thesamet/gerrit/plugins/saml/SamlWebFilter.java:179: error: cannot find symbol

        context.setSessionAttribute(SAML2Client.SAML_RELAY_STATE_ATTRIBUTE, redirectUri);

                                               ^

  symbol:   variable SAML_RELAY_STATE_ATTRIBUTE

  location: class SAML2Client

Target //plugins/gerrit-saml-plugin:gerrit-saml-plugin failed to build

Use --verbose_failures to see the command lines of failed build steps.

ERROR: /u0/mlupo/gerrit/plugins/gerrit-saml-plugin/BUILD:3:1 Building plugins/gerrit-saml-plugin/libgerrit-saml-plugin__plugin-class.jar (3 source files) and running annotation processors (AutoAnnotationProcessor, AutoValueProcessor) failed (Exit 1)

INFO: Elapsed time: 1.053s, Critical Path: 0.64s

INFO: 0 processes.

FAILED: Build did NOT complete successfully

Just for reference: the git log looks like this:

mlupo@prod-gerrit-app04 ~/gerrit/plugins/gerrit-saml-plugin (davidsPRs) $ git log

commit 15b9d3d644b904f9ccec3e78aeb938b1cf7685ff

Author: David Ostrovsky <david@ostrovsky.org>

Date:   Wed Dec 26 19:47:54 2018 +0000

    Fix AnonymousHttpRequest.getHeaderNames method

commit 9122c14edbc0de67c39e72a66a7684b5f884ec3c

Author: David Ostrovsky <david@ostrovsky.org>

Date:   Wed Dec 26 19:34:05 2018 +0000

    Bazel: Add transitive dependencies

commit f9706cc4eafec251ad2b2e60625425fb033e25f4

Author: David Ostrovsky <david@ostrovsky.org>

Date:   Sun Dec 23 19:02:48 2018 +0000

    Implement Bazel build

commit b2cffe647b8ead852f25165d51948ec33907a990

Author: David Ostrovsky <david@ostrovsky.org>

Date:   Tue Dec 25 10:11:23 2018 +0000

    Bump Gerrit plugin API to 2.16.2

commit 949d6becdb0cb3e96342742dfa1dba24c36a100e

Author: David Ostrovsky <david@ostrovsky.org>

Date:   Wed Dec 26 19:40:30 2018 +0000

    Set serviceProviderMetadataResourceFilepath in SAML2 config

   

    This seems to be necessary in new pac4j version. See more details

    in this issue: [1].

   

    [1] https://github.com/pac4j/pac4j/issues/1210

commit c56dc589f13b64d2a70cbf048d0a8fa527bf7b9f

Author: David Ostrovsky <david@ostrovsky.org>

Date:   Sun Dec 23 18:52:15 2018 +0000

    Replace @Nonnull annotation with checkNotNull assertion

commit ab8963738faf5aa829439e945e17d3f916cee9c9

Merge: 16c6b9e dbf220b

Author: Nadav Samet <thesamet@gmail.com>

Date:   Fri Jul 21 09:31:41 2017 -0700

    Merge pull request #13 from arraen/master

   

    Fix maxAuthLifetimeAttr init

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote:

[mike...@aol.com]

A little investigation reveals that

http://www.pac4j.org/apidocs/pac4j/3.2.0/org/pac4j/saml/client/SAML2Client.html has the field: SAML_RELAY_STATE_ATTRIBUTE .

http://www.pac4j.org/apidocs/pac4j/3.3.0/org/pac4j/saml/client/SAML2Client.html does not. Instead, it's moved to SAMLStateGenerator

But you're including pac4j 3.4.0. 
So we need to make some sort of change to  gerrit-saml-plugin/src/main/java/com/thesamet/gerrit/plugins/saml/SamlWebFilter.java

blindly, I tried to change this:

context.setSessionAttribute(SAML2Client.SAML_RELAY_STATE_ATTRIBUTE, redirectUri);

to

context.getSessionStore().set(context, SAML2StateGenerator.SAML_RELAY_STATE_ATTRIBUTE, redirectUri);

It compiles fine.

However, integrating that .jar file into the /gerrit_site/lib and restarting...

I try to log in, our idP does not like it. I'm getting a 400 BAD REQUEST back from the idP.

How did you even get it to compile with pac4j 3.4.0?

Thanks,

Mike

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote:

[David Ostrovsky]

Can it be that you have skipped some of commits included in this series? The one you have missed is here: [1].

[1] https://github.com/thesamet/gerrit-saml-plugin/commit/e9565bf6ef2e61e98e71cc4627c7277e2c89a906

[mike...@aol.com]

David, that was indeed the problem. I missed a commit. It builds fine now and produces the .jar. 

From the client perspective, there are no errors in the gerrit logs now. 

However, our SAML service isn't liking it. I see this rendered in my browser:

--------------------------

Unable to Respond

The login service was unable to identify a compatible way to respond to the requested application. This is generally due to a misconfiguration on the part of the application and should be reported to the application's support team or owner.

------------------------

Just for completeness: Here are the pertinent sections of my gerrit.config file as well as the git log as it exists now. I don't think I need to reorder anything in the git log, but can if you deem it necessary.

[httpd]

    listenUrl = proxy-https://*:8081/gerrit/

    filterClass = com.thesamet.gerrit.plugins.saml.SamlWebFilter

[saml]

    keystorePath = /gerrit/saml_cert/samlKeystore.jks

    keystorePassword = MaskedPassword

    privateKeyPassword = MaskedPassword

    metadataPath = file://gerrit/saml_cert/prod-idp-metadata-gerrit.xml

    maxAuthLifetime = 86400

git log:

mlupo@prod-gerrit-app04 ~/gerrit-saml-plugin (davidsPRs) $ git log

commit 557495ae72881f6a4f42e62cbe29b9e7f191bd40

Author: David Ostrovsky

Date:   Sun Dec 23 20:23:44 2018 +0000

    Bump pac4j-saml to 3.4.0

    

    Require Java 8. Given that this pac4j-saml release depends

    on opensaml library published on Shibboleth repository, add

    this repository to sbt resolver.

commit 15b9d3d644b904f9ccec3e78aeb938b1cf7685ff

Author: David Ostrovsky

Date:   Wed Dec 26 19:47:54 2018 +0000

    Fix AnonymousHttpRequest.getHeaderNames method

commit 9122c14edbc0de67c39e72a66a7684b5f884ec3c

Author: David Ostrovsky

Date:   Wed Dec 26 19:34:05 2018 +0000

    Bazel: Add transitive dependencies

commit f9706cc4eafec251ad2b2e60625425fb033e25f4

Author: David Ostrovsky

Date:   Sun Dec 23 19:02:48 2018 +0000

    Implement Bazel build

commit b2cffe647b8ead852f25165d51948ec33907a990

Author: David Ostrovsky 

Date:   Tue Dec 25 10:11:23 2018 +0000

Date:   Tue Dec 25 10:11:23 2018 +0000

    Bump Gerrit plugin API to 2.16.2

commit 949d6becdb0cb3e96342742dfa1dba24c36a100e

Author: David Ostrovsky

Date:   Wed Dec 26 19:40:30 2018 +0000

    Set serviceProviderMetadataResourceFilepath in SAML2 config

    

    This seems to be necessary in new pac4j version. See more details

    in this issue: [1].

    

    [1] https://github.com/pac4j/pac4j/issues/1210

commit c56dc589f13b64d2a70cbf048d0a8fa527bf7b9f

Author: David Ostrovsky

Date:   Sun Dec 23 18:52:15 2018 +0000

    Replace @Nonnull annotation with checkNotNull assertion

commit ab8963738faf5aa829439e945e17d3f916cee9c9

Merge: 16c6b9e dbf220b

Author: Nadav Samet

Date:   Fri Jul 21 09:31:41 2017 -0700

    Merge pull request #13 from arraen/master

    

    Fix maxAuthLifetimeAttr init

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote:

[mike...@aol.com]

missed one pertinent section of my gerrit.config

[auth]

    type = HTTP

    logoutUrl = https://gerrit.mycompany.com/logout

    httpHeader = X-SAML-UserName

    httpDisplaynameHeader = X-SAML-DisplayName

    httpEmailHeader = X-SAML-EmailHeader

    httpExternalIdHeader = X-SAML-ExternalId

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote:

[mike...@aol.com]

Update: I've been working with our corporate SAML idP folks. The problem is that the gerrit-saml-plugin has changed the idP metadata that we see in /gerrit/file:/gerrit/saml_cert/.  Noting that /gerrit is gerrit's home directory.

New and unexpected:

HTTP-POST" Location="https://gerrit.mycompany.com/gerrit/plugins/gerrit-saml-plugin/saml?client_name=SAML2Client&logoutendpoint=true"/

Original and expected:

HTTP-POST" Location=  "https://gerrit.mycompany.com/gerrit/plugins/gerrit-saml-plugin/saml"

David, was that a change made by you in the plugin?

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote:

[mike...@aol.com]

David,

Per your ask, I will summarize my findings in regards to your generous code changes to the gerrit-saml-plugin:

1. Bazel compilation is wonderfully better than sbt. Ever see a guy do the happy-dance? I surely was. Thank you for this.
   
2. The updates to the code itself required a change on the server-side because of the unexpected parameters on the Location header. in [1]. Unsure if this is a side-effect of updating pac4j-saml to 3.4.0 or if it was due to the headers change you made.
   
3. Once the server side changes were made, SSO is working again with our gerrit instance at 2.16.2. Some users had to clear their browser cache else they received a "server error".
   
4. You need to update the build instructions and perhaps include a link to compiled binary (.jar) in the .md file. You might include a release note about the new parameters that are POSTed with the idP metadata. Especially because it may require coordinating a change on the server side.
   
5. If there is an override to that last point, we might want to add that to the gerrit.config [saml] section...and make appropriate changes to the plugin to read it and use it in the SamlWebFilter.java file. I didn't see anything glaring in the pac4j-saml api to that affect.
   
6. In my experience I've found that Nadaf Samet is slow to address any PRs or asks with the repo. You may wish to move the code to the google code repository on your own. He might appreciate a gerrit maintainer taking ownership and maturing it as you have in your PRs.
   

If you have any questions or concerns please don't hesitate to ask.

[1] HTTP-POST" Location="https://gerrit.mycompany.com/gerrit/plugins/gerrit-saml-plugin/saml?client_name=SAML2Client&logoutendpoint=true"/

Thanks again,

Mike

On Saturday, December 29, 2018 at 12:51:02 PM UTC-5, Mike Lupo wrote: