2.14.7 update to 2.15.12

[Ray Pelkey]

After updating to 2.15.12 from 2.14.7, I get this error when trying to view changes.

'is:wip' operator is not supported by change index version

I chose to do online migration by adding this to the Gerrit config before restarting:

[noteDb "changes"]
        autoMigrate = true

Do we need to wait till the change migrations from reviewdb to notedb has completed?

I thought the auto migrate also did a full online reindex,  is an offline index required afterall for 2.15.12?

Can someone shed some light please?

[Sven Selberg]

Is it just "is:wip" or is it a combined query like "owner:self is:wip"?

Other predicates exclude wip so "owner:self" is in effect "owner:self and not is:wip" so the query above would be "(owner:self and not is:wip) and is:wip" which is the null set.

/Sven

[Ray Pelkey]

Hi Sven,

Thx for the reply.

I'm just select "My" off the menu in the UI, so no other query beyond just scanning for all changes.  I'm trying an offline index now, but I see a lot of

 Missing unknown 4e7e3af78aa6166c5e170c3d3649c8d93ccf7777 errors

I thought the automigrate would do that?

[Ray Pelkey]

just some more details..

if I look at All changes (opened/merged/abandoned) in the UI, that works fine,  it's only My -> Changes that shows this error from the UI.  I stopped the offline index and am trying to resume the online / automgrate to NoteDB.  Perhaps that just needs to finish. will let this run for a few hours and see what happens.

[JR ibkr]

I am getting the same error.

[Ray Pelkey]

you can check the error log to see when the NoteDB migration and the Online Re-Index completes,  that message should go away once these back ground tasks are done.

if you have a large site, this could take some hours.

[JR ibkr]

Ray, I can see following in error.log

ERROR com.google.gerrit.server.inde.OnlineReindexer : Online rindex of changes schema version 48 failed. Succcessfully indexed 0, failed to index 0

[JR ibkr]

Additionally, I see 

[OnlineNoteDbMigrator] ERROR com.google.gerrit.server.notedb.rebuild.NoteDbMigrator : Error Migrating primary storage for XXXX

[Ray Pelkey]

So I believe what should happen is the notedb migration should run before the re-index.

I've had online re-index fail before, but a gerrit restart, should restart that index.  There could be an issue with changes to a project that you may have to work thru resolving tho.

In all the testing I've done, I don't think I've ever seen the notedb migration fail.  are there other errors printed to the log after the Error?  may be some hints there.

[David Ostrovsky]

Am Freitag, 4. Oktober 2019 20:18:20 UTC+2 schrieb JR ibkr:

Additionally, I see 

[OnlineNoteDbMigrator] ERROR com.google.gerrit.server.notedb.rebuild.NoteDbMigrator : Error Migrating primary storage for XXXX

Can you check, if the repository for those changes were removed?

In that case the changes cannot be migrated. You would need to

remove those changes from the ReviewDb first. See this attempt to

fix it, that was staled and git abandoned: [1].

* [1] https://gerrit-review.googlesource.com/c/gerrit/+/235928

 

[JR ibkr]

I was able to circumvent the issue by performing following steps after upgrading to gerrit 2.15:

• activate change index
• running reindex (this took huge amount of time)
• restart gerrit

Downside of this approach was:

• Some code changes were not indexed with following exception 

[Index-Batch-1] ERROR com.google.gerrit.index.Schema : error getting field exactauthor of ChangeData{Change{xxx (xxx), dest=xxx,refs/heads/xxx, status=n}}

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.MissingObjectException: Missing Unknown XXXX

I ran "select note_db_state, count(1) from changes group by note_db_state" and it seems most of the code changes has note_db_state=N and some has note_db_state=NULL. Don't know exactly what it means.

[JR ibkr]

It took me more than 12hours to finish offline migration. Can not use that procedure in production environment. However, I identified all the orphan changes followed new procedure:

• Stop gerrit 2.14
• Delete orphan changes from reviewdb
• init gerrit 2.15
• start gerrit 2.15

In error.log, I see following for all of the changes in changes database.

ERROR com.google.gerrit.server.notedb.rebuild.NoteDbMigrator : Error Migrating primary storage for XXXX

com.google.gerrit.server.notedb.PrimaryStorageMigrator$NoNoteDbStateException: change XXX has no note_db_state; rebuild it first

I thought that in 2.15 online migration would take care of it. It looks like i was wrong. 

[luca.mi...@gmail.com]

Sent from my iPhone

On 8 Oct 2019, at 15:02, JR ibkr <jiga...@gmail.com> wrote:



It took me more than 12hours to finish offline migration. Can not use that procedure in production environment.

There are ways to avoid a 12h outage for the migration. Are you coming to the User Summit in Sunnyvale? I will explain the details on how to do it.

We had only a 5m read only window for that upgrade on GerritHub, with 40k repos.

Off line reindex is not the best way to go

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d495af0b-0600-47c8-9a41-569cfc2a8af3%40googlegroups.com.

[Ray Pelkey]

we've used on line reindex, as well as online notedb migration.  Agree, a 12 hour off line index would be a tough sell for production updates.

[Luca Milanesio]

On 8 Oct 2019, at 15:30, Ray Pelkey <ray.p...@here.com> wrote:

we've used on line reindex, as well as online notedb migration.  Agree, a 12 hour off line index would be a tough sell for production updates.

Two suggestions:

1) *DO NOT* migrate to NoteDb in v2.15: there is a P0 pot-hole that you may hit and then be stuck (see [1]).

2) *DO NOT* reindex all changes at once: of the 12h, 11h 55' are spent reindexing closed changes, you can do that *before* your cutover.

If you follow the above suggestions, you can migrate to v2.15 with a 5' cutover, where you could even keep the system up-and-running if you have HA in place.

HTH

Luca.

[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=10239

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/6e623a8c-7a22-4f8f-bae5-365d3f519923%40googlegroups.com.

[Ray Pelkey]

Hi Luca

could you clarify this just a bit please?  We have a big data base (12gb) and about 10k projects, and in the testing I've performed, the NoteDB, while it takes some hours (about 6) it does not fail.

The installation will drop tables from the database towards the end.  To start the online migration, I've just added the auto migrate settings to the  gerrit config.

your suggesting to not do this?

How does this all work if you do not add the config entries to migrate to noteDB after the installation has already changed the Database tables?  The documentation doesn't really seem to be clear here. 

There's a work in progress option that in testing, I have not opted to use.  Will that continue to use the external DB (In our case postgres)

On Tuesday, October 8, 2019 at 11:02:50 AM UTC-4, lucamilanesio wrote:

To unsubscribe, email rep...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

[Luca Milanesio]

On 8 Oct 2019, at 16:29, Ray Pelkey <ray.p...@here.com> wrote:

Hi Luca

could you clarify this just a bit please?

There will be 1h of clarification next month in Sunnyvale :-)

The topic of migration is a long one, as Gerrit is a stateful service and between v2.14 and v3.0 there are many radical architectural changes in the architecture.

  We have a big data base (12gb) and about 10k projects, and in the testing I've performed, the NoteDB, while it takes some hours (about 6) it does not fail.

Do not migrate to NoteDb in v2.15, it is NOT mandatory and it would potentially cause problems when you will then migrate to v2.16.

(see the bug [1] I've mentioned, it's an accepted P0 without any resolution)

You can stay on ReviewDb in v2.15 and, once you'll have migrated to v2.16, convert to NoteDb afterwards.

The installation will drop tables from the database towards the end.  To start the online migration, I've just added the auto migrate settings to the  gerrit config.

your suggesting to not do this?

Exactly, see above the rationale behind it.

How does this all work if you do not add the config entries to migrate to noteDB after the installation has already changed the Database tables?  The documentation doesn't really seem to be clear here. 

There's a work in progress option that in testing, I have not opted to use.  Will that continue to use the external DB (In our case postgres)

Just keep on using ReviewDb in v2.15, only perform index on-line migration, and that's it.

HTH

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a8dbabdf-22d9-4394-aee6-b51be6dfa819%40googlegroups.com.

[Matthias Sohn]

On Tue, Oct 8, 2019 at 5:35 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 8 Oct 2019, at 16:29, Ray Pelkey <ray.p...@here.com> wrote:

Hi Luca

could you clarify this just a bit please?

There will be 1h of clarification next month in Sunnyvale :-)

The topic of migration is a long one, as Gerrit is a stateful service and between v2.14 and v3.0 there are many radical architectural changes in the architecture.

  We have a big data base (12gb) and about 10k projects, and in the testing I've performed, the NoteDB, while it takes some hours (about 6) it does not fail.

Do not migrate to NoteDb in v2.15, it is NOT mandatory and it would potentially cause problems when you will then migrate to v2.16.

(see the bug [1] I've mentioned, it's an accepted P0 without any resolution)

I think we should amend the 2.15 release notes [2] and clearly mention this to prevent users from falling into this trap.

And we should also mention there how to best do reindexing to avoid long downtimes.

[2] https://www.gerritcodereview.com/2.15.html#notedb-for-changes

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/084199ED-995E-48FA-B7E8-D3AADE375DF9%40gmail.com.

[Luca Milanesio]

On 8 Oct 2019, at 16:43, Matthias Sohn <matthi...@gmail.com> wrote:

On Tue, Oct 8, 2019 at 5:35 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 8 Oct 2019, at 16:29, Ray Pelkey <ray.p...@here.com> wrote:

Hi Luca

could you clarify this just a bit please?

There will be 1h of clarification next month in Sunnyvale :-)

The topic of migration is a long one, as Gerrit is a stateful service and between v2.14 and v3.0 there are many radical architectural changes in the architecture.

  We have a big data base (12gb) and about 10k projects, and in the testing I've performed, the NoteDB, while it takes some hours (about 6) it does not fail.

Do not migrate to NoteDb in v2.15, it is NOT mandatory and it would potentially cause problems when you will then migrate to v2.16.

(see the bug [1] I've mentioned, it's an accepted P0 without any resolution)

I think we should amend the 2.15 release notes [2] and clearly mention this to prevent users from falling into this trap.

And we should also mention there how to best do reindexing to avoid long downtimes.

[2] https://www.gerritcodereview.com/2.15.html#notedb-for-changes

I did it already with [3], in the release notes at [4].

Copying here the relevant section:

"NoteDb for change metadata is considered reasonably stable, and new sites use it by default. However, because of Issue 10239 it is recommended for existing sites to stay on ReviewDb and migrate to NoteDb only after the upgrade to 2.16."

Maybe isn't prominent enough?

Luca.

[3] https://gerrit-review.googlesource.com/c/homepage/+/235156

[4] https://gerrit-review.googlesource.com/c/homepage/+/235156

[Ray Pelkey]

ha! wont be able to be in Sunnyvale,  thank you for clarifying, this is very helpful for us.

this is the spot during the upgrade

Migrate draft changes to private changes (default is work-in-progress) [y/N]?
Replace draft changes with work_in_progress changes ...
done

but I don't think that is relevant no that I review this.

I guess to circumvent the NoteDB migration we just wont add:

[noteDb "changes"]

        autoMigrate = true

to our gerrit config.  and I guess all that allows us to continue to use reviewdb

and to reply to Matthias

I think we should amend the 2.15 release notes [2] and clearly mention this to prevent users from falling into this trap.

And we should also mention there how to best do reindexing to avoid long downtimes.

yes, agree, the doc seems very unclear here, and it's quite easy to follow along with the auto migration course.

On Tuesday, October 8, 2019 at 11:35:17 AM UTC-4, lucamilanesio wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a8dbabdf-22d9-4394-aee6-b51be6dfa819%40googlegroups.com.

[Matthias Sohn]

On Tue, Oct 8, 2019 at 5:50 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 8 Oct 2019, at 16:43, Matthias Sohn <matthi...@gmail.com> wrote:

On Tue, Oct 8, 2019 at 5:35 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 8 Oct 2019, at 16:29, Ray Pelkey <ray.p...@here.com> wrote:

Hi Luca

could you clarify this just a bit please?

There will be 1h of clarification next month in Sunnyvale :-)

The topic of migration is a long one, as Gerrit is a stateful service and between v2.14 and v3.0 there are many radical architectural changes in the architecture.

  We have a big data base (12gb) and about 10k projects, and in the testing I've performed, the NoteDB, while it takes some hours (about 6) it does not fail.

Do not migrate to NoteDb in v2.15, it is NOT mandatory and it would potentially cause problems when you will then migrate to v2.16.

(see the bug [1] I've mentioned, it's an accepted P0 without any resolution)

I think we should amend the 2.15 release notes [2] and clearly mention this to prevent users from falling into this trap.

And we should also mention there how to best do reindexing to avoid long downtimes.

[2] https://www.gerritcodereview.com/2.15.html#notedb-for-changes

I did it already with [3], in the release notes at [4].

Copying here the relevant section:

"NoteDb for change metadata is considered reasonably stable, and new sites use it by default. However, because of Issue 10239 it is recommended for existing sites to stay on ReviewDb and migrate to NoteDb only after the upgrade to 2.16."

Maybe isn't prominent enough?

I would move this under "Important Notes" [2] since I would expect those planning an upgrade at least read this section.

And the following statements given under [2] contradicts the recommendation given in [4]

"With this release, the new Gerrit storage backend, NoteDb, is officially supported for storing change metadata and is the default storage backend for new installations.
For existing sites, migration to the new backend may be done either offline or online in a running server."

if we don't recommend to migrate changes to notedb then we should not state it's officially supported.

Instead we should state that this is experimental in 2.15 and leads to problems when upgrading to 2.16

and that users should defer this migration to 2.16. 

[luca.mi...@gmail.com]

Sent from my iPhone

On 8 Oct 2019, at 17:00, Matthias Sohn <matthi...@gmail.com> wrote:



On Tue, Oct 8, 2019 at 5:50 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 8 Oct 2019, at 16:43, Matthias Sohn <matthi...@gmail.com> wrote:

On Tue, Oct 8, 2019 at 5:35 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 8 Oct 2019, at 16:29, Ray Pelkey <ray.p...@here.com> wrote:

Hi Luca

could you clarify this just a bit please?

There will be 1h of clarification next month in Sunnyvale :-)

The topic of migration is a long one, as Gerrit is a stateful service and between v2.14 and v3.0 there are many radical architectural changes in the architecture.

  We have a big data base (12gb) and about 10k projects, and in the testing I've performed, the NoteDB, while it takes some hours (about 6) it does not fail.

Do not migrate to NoteDb in v2.15, it is NOT mandatory and it would potentially cause problems when you will then migrate to v2.16.

(see the bug [1] I've mentioned, it's an accepted P0 without any resolution)

I think we should amend the 2.15 release notes [2] and clearly mention this to prevent users from falling into this trap.

And we should also mention there how to best do reindexing to avoid long downtimes.

[2] https://www.gerritcodereview.com/2.15.html#notedb-for-changes

I did it already with [3], in the release notes at [4].

Copying here the relevant section:

"NoteDb for change metadata is considered reasonably stable, and new sites use it by default. However, because of Issue 10239 it is recommended for existing sites to stay on ReviewDb and migrate to NoteDb only after the upgrade to 2.16."

Maybe isn't prominent enough?

I would move this under "Important Notes" [2] since I would expect those planning an upgrade at least read this section.

And the following statements given under [2] contradicts the recommendation given in [4]

"With this release, the new Gerrit storage backend, NoteDb, is officially supported for storing change metadata and is the default storage backend for new installations.
For existing sites, migration to the new backend may be done either offline or online in a running server."

if we don't recommend to migrate changes to notedb then we should not state it's officially supported.

True, we can’t say we support NoteDB if there is a P0 that nobody is willing to fix :-(

Instead we should state that this is experimental in 2.15 and leads to problems when upgrading to 2.16

and that users should defer this migration to 2.16. 

Agreed.

Luca

[Luca Milanesio]

Done with:

https://gerrit-review.googlesource.com/c/homepage/+/240055

[Luca Milanesio]

On 8 Oct 2019, at 16:56, Ray Pelkey <ray.p...@here.com> wrote:

ha! wont be able to be in Sunnyvale,  thank you for clarifying, this is very helpful for us.

Recordings will be available anyway, and I am trying to setup a remote live session.

this is the spot during the upgrade

Migrate draft changes to private changes (default is work-in-progress) [y/N]?
Replace draft changes with work_in_progress changes ...
done

but I don't think that is relevant no that I review this.

I guess to circumvent the NoteDB migration we just wont add:

[noteDb "changes"]

        autoMigrate = true

Yes, just keep the configuration as-is and you'll stay on ReviewDb.

to our gerrit config.  and I guess all that allows us to continue to use reviewdb

and to reply to Matthias

I think we should amend the 2.15 release notes [2] and clearly mention this to prevent users from falling into this trap.

And we should also mention there how to best do reindexing to avoid long downtimes.

yes, agree, the doc seems very unclear here, and it's quite easy to follow along with the auto migration course.

Done.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/33dd39fc-a188-4783-b7e1-3e9b58091153%40googlegroups.com.

[Ray Pelkey]

thank you for all the replies Luca,  this is very helpful.  Cheers!

On Tuesday, October 8, 2019 at 1:22:07 PM UTC-4, lucamilanesio wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/33dd39fc-a188-4783-b7e1-3e9b58091153%40googlegroups.com.

[JR ibkr]

Is the following procedure to migrate from 2.14 to 2.15 sounds correct:

1. stop gerrit 2.14
2. go to etc/gerrit.config and add 
• [noteDb "changes"] 
• autoMigrate=False
3. remove orphan changes from database

1. init gerrit 2.15
2. start gerrit 2.15

Gerrit upgraded to 2.15 but when I click "My -> changes", it shows 'is:wip' operator is not supported by change index version.

Did I miss something?

[Luca Milanesio]

On 8 Oct 2019, at 20:36, JR ibkr <jiga...@gmail.com> wrote:

Is the following procedure to migrate from 2.14 to 2.15 sounds correct:

1. stop gerrit 2.14
2. go to etc/gerrit.config and add 
• [noteDb "changes"] 
• autoMigrate=False

Step 2. is optional, migration to NoteDb is *disabled by default*.

3. remove orphan changes from database

You can do this as step 0: you should get rid of the orphans already in v2.14.

4. init gerrit 2.15
5. start gerrit 2.15

Yep.

Gerrit upgraded to 2.15 but when I click "My -> changes", it shows 'is:wip' operator is not supported by change index version.

That's perfectly normal, because the index is about to be re-generated online. Once the online reindex is complete, the error will disappear.

P.S. You could fire a bug on gerritcodereview.com/issues.html for that, the backend API should fail more gracefully and avoid the annoying error message.

Online reindex make take a few hours, and you don't want people asking and panicking for that.

HTH

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/69777cff-b47a-46e5-8252-fea2044696a9%40googlegroups.com.

[Ray Pelkey]

if orphaned means abandoned,,  we have hundreds,  we expire them after 90 days,  perhaps we should tighten that window leading up to our update.

To unsubscribe, email rep...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/69777cff-b47a-46e5-8252-fea2044696a9%40googlegroups.com.

[JR ibkr]

Orphaned code changes= project has been deleted but changes in reviewdb was not

[Ray Pelkey]

revisiting this thread with a question re: removing orphan changes,  not being all that expert in postgres, can anyone provide the psql cmds to perform this step?