Is NoteDB good enough for production use?

[lucamilanesio]

Hi all (and specifically Dave),

I was wondering if the current status of NoteDB on master is good enough (even if not perfect) for a production use.

Do we have NoteDB enabled on gerrit-review.googlesource.com?

Luca.

[Dave Borowitz]

No.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luca Milanesio]

And what about how to enable it just to play with it?

What I’ve done is:

a) Set in gerrit.config:

[gerrit]

        noteDbPath = notedb

[notedb "changes"]

        read = true

        write = true

b) Execute:

java -jar bin/gerrit.war RebuildNotedb

c) Start Gerrit

Anything missing?

Luca.

[Dave Borowitz]

I think that's enough to try it out.

It should mostly work, except that nobody actually does that so there might be lurking bugs. Let me know what you find out :)

I say it's not ready for production use because of both potential bugs and performance issues.

[Luca Milanesio]

The fact of storing the review history into a Git repo is a huge plus for one of my customers … and they would be willing to give it a try on a staging environment.

The rationale behind is:

- a project may last for many years, including maintenance

- review history is key from a tracing and compliance perspective

- having a self-contained and self-describing format inside the Git repo is a key feature for future-proof portability

… and I wouldn’t agree more :-)

Luca.

[Dave Borowitz]

On Tue, Apr 28, 2015 at 12:25 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

The fact of storing the review history into a Git repo is a huge plus for one of my customers … and they would be willing to give it a try on a staging environment.

Sure. Go for it :)

One big caveat is until this is released, the review history will need to be completely rewritten many times.

[Luca Milanesio]

On 28 Apr 2015, at 20:30, Dave Borowitz <dbor...@google.com> wrote:

On Tue, Apr 28, 2015 at 12:25 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

The fact of storing the review history into a Git repo is a huge plus for one of my customers … and they would be willing to give it a try on a staging environment.

Sure. Go for it :)

One big caveat is until this is released, the review history will need to be completely rewritten many times.

Sure, it goes “side-by-side” with the DB so that you can “rebuild” the notes-db with the command mentioned.

For sure with a massive production DB is not feasible, but for playing with it with a staging environment is OK.

Let’s say that they like it, how far are we from a stable release?

Weeks? Months? Years?

Luca.

[Dave Borowitz]

On Tue, Apr 28, 2015 at 12:34 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 28 Apr 2015, at 20:30, Dave Borowitz <dbor...@google.com> wrote:

On Tue, Apr 28, 2015 at 12:25 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

The fact of storing the review history into a Git repo is a huge plus for one of my customers … and they would be willing to give it a try on a staging environment.

Sure. Go for it :)

One big caveat is until this is released, the review history will need to be completely rewritten many times.

Sure, it goes “side-by-side” with the DB so that you can “rebuild” the notes-db with the command mentioned.

For sure with a massive production DB is not feasible, but for playing with it with a staging environment is OK.

Let’s say that they like it, how far are we from a stable release?

Weeks? Months? Years?

Months. I really want to finish in 2015. More practically speaking, we might only get changes migrated in 2015.

[Björn Pedersen]

Hi,

is there any update on notedb usability/uses?

Björn

[Dave Borowitz]

Haha well we didn't finish in 2015, but we're really nearly there for changes.

I think it is stable enough for production-ish use now.

Some caveats:

-I have no idea if the performance of RebuildNoteDb is acceptable (but I'd love people to try it out).

-I think performance of individual NoteDb reads/writes is probably acceptable, but I have no data on running it in a loaded server.

-We don't have the code to turn off writes to the ReviewDb backend, and we're still a few months away from removing the database dependency. Which for you right now is actually a good thing, because it means you can always switch back to ReviewDb with no data loss.

-I expect there to still be some bugs, which we will fix quickly, but this does mean you have to be willing to run a master build and upgrade as necessary.

We hope to convert gerrit-review in the next couple of weeks. (What's holding us back is not stability of the NoteDb code so much as the complexity of a no-downtime conversion, which is not applicable to anyone else.)

[Björn Pedersen]

Thanks for the answers (and the work on gerrit of course).

I did some first tests:
 * performance of RebuildNoteDb seems acceptable for us (a few seconds for our largest repo (~10K changes)
 * the delete-project is incompatible:

[2016-04-18 17:35:33,775] [HTTP-94] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /projects/frm2%2Fnicos%2Fn
icos-core4/deleteproject~delete
java.lang.ClassCastException: com.google.gerrit.server.schema.DisabledChangesReviewDbWrapper cannot be cast to com.google.gwtorm.jd
bc.JdbcSchema
        at com.googlesource.gerrit.plugins.deleteproject.database.DatabaseDeleteHandler.delete(DatabaseDeleteHandler.java:80)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteProject.doDelete(DeleteProject.java:148)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteAction.doDelete(DeleteAction.java:31)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteProject.apply(DeleteProject.java:104)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteAction.apply(DeleteAction.java:31)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteProject.apply(DeleteProject.java:49)

[Dave Borowitz]

On Mon, Apr 18, 2016 at 12:06 PM, 'Björn Pedersen' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Thanks for the answers (and the work on gerrit of course).

I did some first tests:
 * performance of RebuildNoteDb seems acceptable for us (a few seconds for our largest repo (~10K changes)

Good to hear. Though that's almost a little surprisingly fast...did you verify that it's created all the /meta refs for each change? Or maybe you're just running this on an extremely beefy machine.

 

 * the delete-project is incompatible:

[2016-04-18 17:35:33,775] [HTTP-94] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /projects/frm2%2Fnicos%2Fn
icos-core4/deleteproject~delete
java.lang.ClassCastException: com.google.gerrit.server.schema.DisabledChangesReviewDbWrapper cannot be cast to com.google.gwtorm.jd
bc.JdbcSchema
        at com.googlesource.gerrit.plugins.deleteproject.database.DatabaseDeleteHandler.delete(DatabaseDeleteHandler.java:80)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteProject.doDelete(DeleteProject.java:148)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteAction.doDelete(DeleteAction.java:31)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteProject.apply(DeleteProject.java:104)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteAction.apply(DeleteAction.java:31)
        at com.googlesource.gerrit.plugins.deleteproject.DeleteProject.apply(DeleteProject.java:49)

I think for this specific usage you can reimplement one of the copies of unwrapDb we have scattered throughout the code. I have intentionally not put that in a common library because I don't don't want people to call it without thinking carefully first.

[Björn Pedersen]

Hi,

with https://gerrit-review.googlesource.com/#/q/status:open+project:gerrit+branch:master+topic:notedb-fixes applied,
I only have one error left (on to changes in my case):

The affected changes both have a comment on the base submitted from a non-current patchset (and where imported using importer-plugin from another instance).

Stacktrace:

[2016-04-21 09:31:07,145] [RebuildChange-4] ERROR com.google.gerrit.pgm.RebuildNoteDb : Failed to rebuild change 4350
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: All comments being added must all have the same PatchSet.Id. The comment below does not have the same PatchSet.Id as the others (4350,14).
PatchLineComment{key=4350,19,custom/antares/setups/startup.py,da2a55be_0c0a57d2,lineNbr=0,author=1000038,writtenOn=2013-07-17 10:24:10.963,status=P,side=0,message=Please keep this and remove special/startup.py,parentUuid=,range=,revId=8d91c1dd2cc7c931bebbdd557ab25242d1ab49f7tag=}
...
Caused by: java.lang.IllegalArgumentException: All comments being added must all have the same PatchSet.Id. The comment below does not have the same PatchSet.Id as the others (4350,14).
PatchLineComment{key=4350,19,custom/antares/setups/startup.py,da2a55be_0c0a57d2,lineNbr=0,author=1000038,writtenOn=2013-07-17 10:24:10.963,status=P,side=0,message=Please keep this and remove special/startup.py,parentUuid=,range=,revId=8d91c1dd2cc7c931bebbdd557ab25242d1ab49f7tag=}
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:146)
        at com.google.gerrit.server.notedb.ChangeNoteUtil.buildNote(ChangeNoteUtil.java:472)
        at com.google.gerrit.server.notedb.RevisionNoteBuilder.build(RevisionNoteBuilder.java:117)
        at com.google.gerrit.server.notedb.ChangeUpdate.storeRevisionNotes(ChangeUpdate.java:414)
        at com.google.gerrit.server.notedb.ChangeUpdate.applyImpl(ChangeUpdate.java:596)
        at com.google.gerrit.server.notedb.AbstractChangeUpdate.apply(AbstractChangeUpdate.java:185)
        at com.google.gerrit.server.notedb.NoteDbUpdateManager.addUpdates(NoteDbUpdateManager.java:347)
        at com.google.gerrit.server.notedb.NoteDbUpdateManager.addCommands(NoteDbUpdateManager.java:324)
        at com.google.gerrit.server.notedb.NoteDbUpdateManager.stage(NoteDbUpdateManager.java:225)
        at com.google.gerrit.server.notedb.ChangeRebuilderImpl.execute(ChangeRebuilderImpl.java:163)
        at com.google.gerrit.server.notedb.ChangeRebuilderImpl.rebuild(ChangeRebuilderImpl.java:147)
        at com.google.gerrit.server.notedb.ChangeRebuilder$1.call(ChangeRebuilder.java:43)
        at com.google.gerrit.server.notedb.ChangeRebuilder$1.call(ChangeRebuilder.java:39)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)

[Dave Borowitz]

Yep I'm seeing that error too. Thanks for taking a look at the specific case where that happens.

I also have a change at the end of the series that should hopefully improve performance on less-beefy machines, I'll try to get that out today as well.

[Dave Borowitz]

That error should hopefully be fixed by https://gerrit-review.googlesource.com/76848

[Dave Borowitz]

Well, the first patch set is broken, but you get the idea, I'll fix it.

[Dave Borowitz]

Ok https://gerrit-review.googlesource.com/76848 should be good to go now. It's chugging through my copy of AOSP. I hope for your sake your repos are not on a spinning disk, it's kind of painful.

[Joan G]

Hi,

What are the config to enable NoteDB as of today?

Cheers

[Joan G]

Is this a dead discussion?

[Dave Borowitz]

Not at all, see https://gerrit-review.googlesource.com/98070 :)

I would have responded last week, but I wanted to write that doc first.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Joan G]

Great! Thanks Dave Borowitz, that's a very good document summing up all the work on NoteDB indeed.

Do you know if, according to this document: https://gerrit.googlesource.com/plugins/multi-master/+/master/src/main/resources/Documentation/config.md

and once fully on NoteDB, we will be able to run a multi-master just by starting instances on a shared NFS?

I expect this to work as git is an immutable, append-only storage. I'm I right?

Cheers

[Dave Borowitz]

NoteDb is not really related to multi-master (although it will eventually make multi-site replication saner). You could do what you're describing today as long as you use a database server and not H2. But there are other things unrelated to ReviewDb/NoteDb that make this not a good idea, I forget exactly what they are, but you can probably find something in the list archives.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Joan G]

"as long as you use a database server and not H2"

This is what bothers us, maintaining a PostgreSQL on the side, with NoteDB we should be able to just have multiple Gerrit docker instances running on Kubernetes that all shares a simple NFS and nothing else to maintain :)

Really looking forward to this full NoteDB support!

Thanks guys

[Dave Borowitz]

Yep, file that under "simplicity" :)

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.