Migration to 2.15-rc2 from 2.13.9 does not work

[Dariusz Luksza]

Direct migration from Gerrit 2.13.9 to 2.15-rc2 doesn't work, because
of missing column 'status' of 'accounts' table which is used in select
statement in Schema 154. The same issue occurs when fresh 2.13.9
install is being updated to 2.15-rc2.

It seams that 'status' column should be added by schema 141 (which is
included in Gerrit 2.14). But somehow this column is not created when
going directly from schema 130 to 161.

Here is the exception:

Exception in thread "main" com.google.gwtorm.server.OrmException:
Cannot upgrade schema
at com.google.gerrit.server.schema.SchemaUpdater.update(SchemaUpdater.java:120)
at com.google.gerrit.pgm.init.BaseInit$SiteRun.upgradeSchema(BaseInit.java:389)
at com.google.gerrit.pgm.init.BaseInit.run(BaseInit.java:146)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:203)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:108)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:63)
at Main.main(Main.java:24)
Caused by: org.h2.jdbc.JdbcSQLException: Column "STATUS" not found;
SQL statement:
SELECT account_id, registered_on, full_name, preferred_email, status,
inactive FROM accounts [42122-176]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
at org.h2.message.DbException.get(DbException.java:178)
at org.h2.message.DbException.get(DbException.java:154)
at org.h2.expression.ExpressionColumn.optimize(ExpressionColumn.java:148)
at org.h2.command.dml.Select.prepare(Select.java:831)
at org.h2.command.Parser.prepareCommand(Parser.java:248)
at org.h2.engine.Session.prepareLocal(Session.java:442)
at org.h2.engine.Session.prepareCommand(Session.java:384)
at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1188)
at org.h2.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:75)
at com.google.gerrit.server.schema.Schema_154.scanAccounts(Schema_154.java:81)
at com.google.gerrit.server.schema.Schema_154.migrateData(Schema_154.java:64)
at com.google.gerrit.server.schema.SchemaVersion.migrateData(SchemaVersion.java:153)
at com.google.gerrit.server.schema.SchemaVersion.upgradeFrom(SchemaVersion.java:92)
at com.google.gerrit.server.schema.SchemaVersion.check(SchemaVersion.java:83)
at com.google.gerrit.server.schema.SchemaUpdater.update(SchemaUpdater.java:118)

--
Best regards

Blog: https://luksza.org http://habitchallenge.co
LinkedIn: https://linkedin.com/in/dariuszluksza

[Steffen Gebert]

Hi,

I can confirm that. This was actually one of the contributing factor, why we migrated https://review.typo3.org only to 2.14 last weekend. I could fix the failing migration by applying the following SQL statement, but it felt just too unstable..

> alter table accounts ADD `status` VARCHAR(255) NULL;

Steffen

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> <console.out.txt>

[Edwin Kempin]

AFAIK that's a limitation of our schema migration framework :(

The schema migration framework knows 2 versions of the schema, the old one that is currently used in ReviewDb and the new one to which the upgrade is done.

Before running the schema migrations it adds all new columns from the new schema to ReviewDb so that the schema migrations have access to the superset of the columns. After the schema migrations are done old columns that no longer exist in the new schema are removed.

If in between the 2 schema versions a column is both added and removed again the schema migration framework has no way of knowing about it and hence cannot provide it for the schema migrations.

No idea how to fix this.

> To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

> <console.out.txt>

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Martin Fick]

On Tuesday, December 12, 2017 05:34:01 PM 'Edwin Kempin' via

Repo and Gerrit Discussion wrote:
> AFAIK that's a limitation of our schema migration
> framework :( The schema migration framework knows 2
> versions of the schema, the old one that is currently
> used in ReviewDb and the new one to which the upgrade is
> done.
> Before running the schema migrations it adds all new
> columns from the new schema to ReviewDb so that the
> schema migrations have access to the superset of the
> columns. After the schema migrations are done old columns
> that no longer exist in the new schema are removed.
>
> If in between the 2 schema versions a column is both added
> and removed again the schema migration framework has no
> way of knowing about it and hence cannot provide it for
> the schema migrations.
>
> No idea how to fix this.

If we can't fix it, should we prevent migrations to 2.15 from
anything but 2.14?

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Luca Milanesio]

On 12 Dec 2017, at 19:06, Martin Fick <mf...@codeaurora.org> wrote:

On Tuesday, December 12, 2017 05:34:01 PM 'Edwin Kempin' via 
Repo and Gerrit Discussion wrote:

AFAIK that's a limitation of our schema migration
framework :( The schema migration framework knows 2
versions of the schema, the old one that is currently
used in ReviewDb and the new one to which the upgrade is
done.
Before running the schema migrations it adds all new
columns from the new schema to ReviewDb so that the
schema migrations have access to the superset of the
columns. After the schema migrations are done old columns
that no longer exist in the new schema are removed.

If in between the 2 schema versions a column is both added
and removed again the schema migration framework has no
way of knowing about it and hence cannot provide it for
the schema migrations.

No idea how to fix this.

If we can't fix it, should we prevent migrations to 2.15 from 
anything but 2.14?

Seems reasonable to me, bearing in mind that 2.15 = NoteDb and it would be best to avoid to support multiple X-to-2.15 migrations

-Martin

-- 
The Qualcomm Innovation Center, Inc. is a member of Code 
Aurora Forum, hosted by The Linux Foundation

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Saša Živkov]

On Tue, Dec 12, 2017 at 8:06 PM, Martin Fick <mf...@codeaurora.org> wrote:

On Tuesday, December 12, 2017 05:34:01 PM 'Edwin Kempin' via
Repo and Gerrit Discussion wrote:
> AFAIK that's a limitation of our schema migration
> framework :( The schema migration framework knows 2
> versions of the schema, the old one that is currently
> used in ReviewDb and the new one to which the upgrade is
> done.
> Before running the schema migrations it adds all new
> columns from the new schema to ReviewDb so that the
> schema migrations have access to the superset of the
> columns. After the schema migrations are done old columns
> that no longer exist in the new schema are removed.
>
> If in between the 2 schema versions a column is both added
> and removed again the schema migration framework has no
> way of knowing about it and hence cannot provide it for
> the schema migrations.
>
> No idea how to fix this.

If we can't fix it, should we prevent migrations to 2.15 from
anything but 2.14?

Yes, I think so. We always used to delete all Schema_NNN versions

starting from which it is impossible to upgrade to the current version.

 

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Jacek Centkowski]

Hi,

Just before we throw the baby out with bathwater can we consider quite simple fix for it:

https://gerrit-review.googlesource.com/#/c/gerrit/+/148031

I understand that some of us are going from version to next version but the others prefer longer jumps (for various reasons ;)).

In addition I would like to point out that AFAIR 2.15 (or 2.16) is the only one that is upgradable from SQL to notedb therefore we would have to have that step in the middle anyway.

Having said that lets not create (if possible of course :D) intermediate steps if not needed (which is the case now ;)).

Regards,

Jacek

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Edwin Kempin]

On Thu, Dec 14, 2017 at 1:00 PM, Jacek Centkowski <geminica...@gmail.com> wrote:

Hi,

Just before we throw the baby out with bathwater can we consider quite simple fix for it:

https://gerrit-review.googlesource.com/#/c/gerrit/+/148031

Nice, thanks for the fix!!

 

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Dave Borowitz]

On Thu, Dec 14, 2017 at 7:00 AM, Jacek Centkowski <geminica...@gmail.com> wrote:

Hi,

Just before we throw the baby out with bathwater can we consider quite simple fix for it:

https://gerrit-review.googlesource.com/#/c/gerrit/+/148031

I understand that some of us are going from version to next version but the others prefer longer jumps (for various reasons ;)).

In addition I would like to point out that AFAIR 2.15 (or 2.16) is the only one that is upgradable from SQL to notedb

Small correction: this is true for the online migration. 3.0 will allow an offline NoteDb migration by running init followed by migrate-to-note-db.

 

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Jacek Centkowski]

On Thursday, 14 December 2017 16:12:18 UTC+1, Dave Borowitz wrote:

Small correction: this is true for the online migration. 3.0 will allow an offline NoteDb migration by running init followed by migrate-to-note-db.

Right - just briefly watched your presentation from GUS 2017:

3.0 is the last one that supports offline migration. Starting from 3.1 however migration is no longer possible (neither online nor offline), right?

IOW if I am not mistaken ;) migration from 2.14 would require offline migration being called from 3.0 and init on 3.1.

Regards

Jacek

 

[Dave Borowitz]

On Thu, Dec 14, 2017 at 10:54 AM, Jacek Centkowski <geminica...@gmail.com> wrote:

On Thursday, 14 December 2017 16:12:18 UTC+1, Dave Borowitz wrote:

Small correction: this is true for the online migration. 3.0 will allow an offline NoteDb migration by running init followed by migrate-to-note-db.

Right - just briefly watched your presentation from GUS 2017:

3.0 is the last one that supports offline migration. Starting from 3.1 however migration is no longer possible (neither online nor offline), right?

Right. 

 

IOW if I am not mistaken ;) migration from 2.14 would require offline migration being called from 3.0 and init on 3.1.

For 2.14 -> 3.1, there are two possible migration paths:

Option 1:

* init @2.15

* offline or online migration @2.15

* init @3.1

Option 2:

* init @3.0

* offline migration @3.0

* init @3.1

If you're already on NoteDb, you can go to 3.1 (or any later 3.x) with only a single "init", regardless of what version you're coming from. In practice, of course, you can only be on NoteDb if you're on 2.15.

 

Regards

Jacek

 

[Jacek Centkowski]

That is precisely why I would like to avoid having 2.14 added to this picture :D

Thanks for detailed explanation!
Regards
Jacek

On Thu, Dec 14, 2017, 21:03 Dave Borowitz <dbor...@google.com> wrote:

On Thu, Dec 14, 2017 at 10:54 AM, Jacek Centkowski <geminica...@gmail.com> wrote:

On Thursday, 14 December 2017 16:12:18 UTC+1, Dave Borowitz wrote:

Small correction: this is true for the online migration. 3.0 will allow an offline NoteDb migration by running init followed by migrate-to-note-db.

Right - just briefly watched your presentation from GUS 2017:

3.0 is the last one that supports offline migration. Starting from 3.1 however migration is no longer possible (neither online nor offline), right?

Right. 

 

IOW if I am not mistaken ;) migration from 2.14 would require offline migration being called from 3.0 and init on 3.1.

For 2.14 -> 3.1, there are two possible migration paths:

Option 1:

* init @2.15

* offline or online migration @2.15

* init @3.1

Option 2:

* init @3.0

* offline migration @3.0

* init @3.1

If you're already on NoteDb, you can go to 3.1 (or any later 3.x) with only a single "init", regardless of what version you're coming from. In practice, of course, you can only be on NoteDb if you're on 2.15.

 

Regards

Jacek

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Jacek Centkowski]

That is precisely why I would like to avoid having 2.14 added to this picture :D

Of course in migration from 2.13 context ;)