Gerrit server goes down when uploading code,then get an error in gerrit log: java.lang.NullPointerException: change 176952 not found in ReviewDb

[zhang....@zte.com.cn]

Our gerrit server goes down when someone uploading the code. 

Now when somebody access the status:open page or upload code to this git repo will get an error:500 Internal server error.

I have tried to insert 5 fake data to these  5 table in DB,but it have another error（org.eclipse.jgit.errors.InvalidObjectIdException: Invalid id: ）,Maybe  they were not enough ?

 public | change_messages             | table    | gerrit2

I guess the code has been uploaded to codereview but not be written into PostgreSQL DB. 

 public | changes                     | table    | gerrit2

 public | patch_comments              | table    | gerrit2

 public | patch_set_approvals         | table    | gerrit2

 public | patch_sets                  | table    | gerrit2

[2016-12-01 18:28:44,644] [HTTP-1012160] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/?q=status:open+is:mergeable+conflicts:177978&O=a

java.lang.NullPointerException: change 176952 not found in ReviewDb

at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:251)

at com.google.gerrit.server.notedb.ChangeNotes$Factory.loadChangeFromDb(ChangeNotes.java:157)

at com.google.gerrit.server.notedb.ChangeNotes$Factory.create(ChangeNotes.java:170)

at com.google.gerrit.server.query.change.ChangeData.notes(ChangeData.java:759)

at com.google.gerrit.server.query.change.ChangeData.reviewers(ChangeData.java:932)

at com.google.gerrit.server.project.ChangeControl.isReviewer(ChangeControl.java:364)

at com.google.gerrit.server.project.ChangeControl.isDraftVisible(ChangeControl.java:445)

at com.google.gerrit.server.project.ChangeControl.isVisible(ChangeControl.java:211)

at com.google.gerrit.server.query.change.ChangeIsVisibleToPredicate.match(ChangeIsVisibleToPredicate.java:67)

at com.google.gerrit.server.query.change.ChangeIsVisibleToPredicate.match(ChangeIsVisibleToPredicate.java:27)

at com.google.gerrit.server.query.AndSource.match(AndSource.java:147)

at com.google.gerrit.server.query.AndSource.readImpl(AndSource.java:101)

at com.google.gerrit.server.query.AndSource.read(AndSource.java:85)

at com.google.gerrit.server.query.QueryProcessor.query(QueryProcessor.java:191)

at com.google.gerrit.server.query.QueryProcessor.query(QueryProcessor.java:139)

at com.google.gerrit.server.query.change.QueryChanges.query(QueryChanges.java:123)

at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:94)

at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:38)

at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:334)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:122)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:60)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:105)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:499)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:744)

[Matthias Sohn]

On Thu, Dec 1, 2016 at 11:51 AM, <zhang....@zte.com.cn> wrote:

Our gerrit server goes down when someone uploading the code. 

Now when somebody access the status:open page or upload code to this git repo will get an error:500 Internal server error.

did you check the error log <gerrit site>/logs/error_log for details about this error ?

I have tried to insert 5 fake data to these  5 table in DB,but it have another error（org.eclipse.jgit.errors.InvalidObjectIdException: Invalid id: ）,Maybe  they were not enough ?

inserting random data into the database will most probably not fix an internal server error

without inspecting the logs what went wrong

 

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[zhang....@zte.com.cn]

Can I delete all files in index folder and reindex to create a new one

在 2016年12月1日星期四 UTC+8下午6:52:22，zhang....@zte.com.cn写道：

[zhang....@zte.com.cn]

Thank you Matthias Sohn

It's the detail:

[2016-12-01 19:10:56,957] [HTTP-1080367] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/?q=status:open+is:mergeable+conflicts:177985&O=a

java.lang.NullPointerException: change 176952 not found in ReviewDb

Everyone who access the front pag of Gerrit will get an error log line like this.

Or I have tried :

ssh -p 29418 10.67.16.29  gerrit review 176952,1 --delete

And also got an error:

fatal: internal server error

在 2016年12月1日星期四 UTC+8下午7:49:17，Matthias Sohn写道：

[zhang....@zte.com.cn]

If I remove the git repository which has problem from gerrit repo folder,the error will disappear. 

And if I create a clean repository with the same name in repo folder, It will also get the error.

 Now I know the information:

change_id  branch  commit hash 

but I don't know how to delete it from  status:open page of gerrit codereview.

[Sven Selberg]

Hi!

If you still have the original repo you can check <repo>/.git/refs/changes/52/176952/ to see if the dir is there and if there are any patchsets in it.

Then you could remove it from the git and do a full reindex [1] of the server. This would require downtime and the change would be lost, but should fix your problem if you didn't corrupt the DB.

/Sven

[1] https://gerrit-review.googlesource.com/Documentation/pgm-reindex.html

---

From: repo-d...@googlegroups.com <repo-d...@googlegroups.com> on behalf of zhang....@zte.com.cn <zhang....@zte.com.cn>
Sent: Thursday, December 1, 2016 1:15 PM
To: Repo and Gerrit Discussion
Subject: Re: Gerrit server goes down when uploading code,then get an error in gerrit log: java.lang.NullPointerException: change 176952 not found in ReviewDb

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[zhang....@zte.com.cn]

Hi Sven Selberg

Thank you for your help.

I don't know why but maybe due to bare repository the folder <repo>/refs/changes is empty.

Should I backup and remove the gerrit index folder before full reindex? And Would I lose all changes data after full reindex?

在 2016年12月1日星期四 UTC+8下午9:55:13，Sven Selberg写道：

[zhang....@zte.com.cn]

Hi all

I've solved this issue by all reindex and upgrade to 2.13.3.

在 2016年12月2日星期五 UTC+8上午9:10:05，zhang....@zte.com.cn写道：