[JIRA] (JENKINS-60796) ERROR: current transaction is aborted, commands ignored until end of transaction block

4 views
Skip to first unread message

dilipm79@gmail.com (JIRA)

unread,
Jan 16, 2020, 10:31:07 AM1/16/20
to jenkinsc...@googlegroups.com
Dilip Mahadevappa created an issue
 
Jenkins / Bug JENKINS-60796
ERROR: current transaction is aborted, commands ignored until end of transaction block
Issue Type: Bug Bug
Assignee: rsandell
Components: gerrit-trigger-plugin
Created: 2020-01-16 15:30
Environment: Jenkins ver. 2.46.2
Gerrit trigger plugin: 2.27.5
Priority: Minor Minor
Reporter: Dilip Mahadevappa

We have got 3 jobs configured to be triggered for the same project. When these jobs starts around the same time (with few ms difference) there is error from Gerrit plugin.

Ex:

2020-01-15 05:39:12.685 GMT-05:00 Triggered by Gerrit: https://gerrit.ericsson.se/6738979
2020-01-15 05:39:12.736 GMT-05:00 ERROR Gerrit response: Server Error

Looks like this is happening due to concurrent HTTP POST requests

Error at Gerrit Server side:

[2020-01-15 13:42:53,225] [HTTP-14255808] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /a/changes/gen2%2Fci-config~main~Id9dd4c734f8f076158772f146dd904ef3f0e408b/revisions/a3e8ee8430931703189d8e13d2544fe715cbb03d/review
com.google.gerrit.server.update.UpdateException: com.google.gwtorm.server.OrmException: fetch failure on patch_sets
	at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:798)
	at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:696)
	at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:670)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:258)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:66)
	at com.google.gerrit.server.update.ReviewDbBatchUpdate.executeChangeOps(ReviewDbBatchUpdate.java:533)
	at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:299)
	at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:426)
	at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:421)
	at com.google.gerrit.server.change.PostReview.apply(PostReview.java:303)
	at com.google.gerrit.server.change.PostReview.apply(PostReview.java:183)
	at com.google.gerrit.server.change.PostReview.apply(PostReview.java:126)
	at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:375)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:451)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:75)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:73)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:111)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RequireIdentifiedUserFilter.doFilter(RequireIdentifiedUserFilter.java:50)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
	at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:67)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:129)
	at com.ericsson.gerrit.plugins.goimport.GoImportFilter.doFilter(GoImportFilter.java:122)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:129)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.ProjectBasicAuthFilter.doFilter(ProjectBasicAuthFilter.java:100)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:72)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.gwtorm.server.OrmException: fetch failure on patch_sets
	at com.google.gwtorm.schema.sql.SqlDialect.convertError(SqlDialect.java:162)
	at com.google.gwtorm.schema.sql.DialectPostgreSQL.convertError(DialectPostgreSQL.java:65)
	at com.google.gwtorm.jdbc.JdbcAccess.convertError(JdbcAccess.java:489)
	at com.google.gwtorm.jdbc.JdbcAccess.queryOne(JdbcAccess.java:137)
	at com.google.gerrit.reviewdb.client.PatchSet_Access_patchSets_GwtOrm$$12.get(Unknown Source)
	at com.google.gerrit.server.PatchSetUtil.get(PatchSetUtil.java:58)
	at com.google.gerrit.server.PatchSetUtil.current(PatchSetUtil.java:53)
	at com.google.gerrit.server.change.PostReviewers$Op.updateChange(PostReviewers.java:425)
	at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:732)
	... 73 more
Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
	at com.google.gwtorm.jdbc.JdbcAccess.queryOne(JdbcAccess.java:119)
	... 78 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "patch_set_approvals_pkey"
  Detail: Key (change_id, patch_set_id, account_id, category_id)=(6739784, 1, 1017701, Verified-Zoovy) already exists.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:481)
	at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:840)
	at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1538)
	at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
	at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
	at com.google.gwtorm.schema.sql.SqlDialect.executeBatch(SqlDialect.java:448)
	at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:460)
	at com.google.gwtorm.jdbc.JdbcAccess.insertAsBatch(JdbcAccess.java:215)
	at com.google.gwtorm.jdbc.JdbcAccess.insert(JdbcAccess.java:166)
	at com.google.gwtorm.jdbc.JdbcAccess.upsert(JdbcAccess.java:338)
	at com.google.gerrit.server.ApprovalsUtil.addReviewers(ApprovalsUtil.java:255)
	at com.google.gerrit.server.ApprovalsUtil.addReviewers(ApprovalsUtil.java:213)
	at com.google.gerrit.server.change.PostReviewers$Op.updateChange(PostReviewers.java:413)
	... 74 more
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

dilipm79@gmail.com (JIRA)

unread,
Jan 16, 2020, 10:36:07 AM1/16/20
to jenkinsc...@googlegroups.com
Dilip Mahadevappa updated an issue
Change By: Dilip Mahadevappa
We have got 3 jobs configured to be triggered for the same project. When these jobs starts around the same time (with few ms difference) there is error from Gerrit plugin.
Ex *Job #1*
2020-01-15 05 : 39:12.680  Triggered by Gerrit: https://gerrit.ericsson.se/6738979
2020-01-15 05:39:14.925  [EnvInject] - Loading node environment variables.
{code }

*Job #2*
{code :java}

2020-01-15 05:39:12.685 GMT-05:00 Triggered by Gerrit: https://gerrit.ericsson.se/6738979
2020-01-15 05:39:12.736
GMT-05:00 ERROR Gerrit response: Server Error
{code}

 There is 5 ms difference between two jobs..

 Looks like this is happening due to concurrent [HTTP POST requests|https://github.com/sonyxperiadev/gerrit-events/blob/bfee4c4dfd1d6dd8a136ce8e349025e888f272c4/src/main/java/com/sonymobile/tools/gerrit/gerritevents/workers/rest/AbstractRestCommandJob2.java#L144]

* Error at Gerrit Server side: *

{code:java}
[2020-01-15 13 05 : 42 39 : 53 12 , 225 733 ] [HTTP- 14255808 13958483 ] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /a/changes/gen2%2Fci-config~main~ Id9dd4c734f8f076158772f146dd904ef3f0e408b Ieb21ba65941b17e9d43cb82069f75c74b4bb636d /revisions/ a3e8ee8430931703189d8e13d2544fe715cbb03d a7e7e5753887b44b63638e516b96b0b5b9d6aa60 /review

com.google.gerrit.server.update.UpdateException: com.google.gwtorm.server.OrmException: fetch failure on patch_sets
at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:798)
at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:696)
at com.google. gerrit common . server util . update concurrent . ReviewDbBatchUpdate TrustedListenableFutureTask $ ChangeTask TrustedFutureInterruptibleTask . call runInterruptibly ( ReviewDbBatchUpdate TrustedListenableFutureTask .java: 670 125 )
at com.google. common gerrit . util server . concurrent update . TrustedListenableFutureTask ReviewDbBatchUpdate $ TrustedFutureInterruptibleTask ChangeTask . runInterruptibly call ( TrustedListenableFutureTask ReviewDbBatchUpdate .java: 125 670 )
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate co key value violates unique constraint "patch_set_approvals_pkey"
  Detail: Key (change_id, patch_set_id, account_id, category_id)=( 6739784 6738979 , 1, 1017701, Verified-Zoovy) already exists.

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:481)
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:840)
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1538)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at com.google.gwtorm.schema.sql.SqlDialect.executeBatch(SqlDialect.java:448)
at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:460)
at com.google.gwtorm.jdbc.JdbcAccess.insertAsBatch(JdbcAccess.java:215)
at com.google.gwtorm.jdbc.JdbcAccess.insert(JdbcAccess.java:166)
at com.google.gwtorm.jdbc.JdbcAccess.upsert(JdbcAccess.java:338)
at com.google.gerrit.server.ApprovalsUtil.addReviewers(ApprovalsUtil.java:255)
at com.google.gerrit.server.ApprovalsUtil.addReviewers(ApprovalsUtil.java:213)
at com.google.gerrit.server.change.PostReviewers$Op.updateChange(PostReviewers.java:413)
... 74 more

 {code}

dilipm79@gmail.com (JIRA)

unread,
Jan 16, 2020, 10:37:06 AM1/16/20
to jenkinsc...@googlegroups.com
Dilip Mahadevappa updated an issue
We have got 3 jobs configured to be triggered for the same project. When these jobs starts around the same time (with few ms difference) there is error from Gerrit plugin.

*Job #1*
 {code:java}
 2020-01-15 05:39:12.680  Triggered by Gerrit: https://gerrit.ericsson.se/6738979

2020-01-15 05:39:14.925  [EnvInject] - Loading node environment variables.
{code}

*Job #2*
{code:java}
2020-01-15 05:39:12.685 GMT-05:00 Triggered by Gerrit: https://gerrit.ericsson.se/6738979
2020-01-15 05:39:12.736  ERROR Gerrit response: Server Error
*Error at Gerrit Server side:*

{code:java}
[2020-01-15 05:39:12,733] [HTTP-13958483] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /a/changes/gen2%2Fci-config~main~Ieb21ba65941b17e9d43cb82069f75c74b4bb636d/revisions/a7e7e5753887b44b63638e516b96b0b5b9d6aa60/review

com.google.gerrit.server.update.UpdateException: com.google.gwtorm.server.OrmException: fetch failure on patch_sets
at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:798)
at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:696)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:670)
Caused by: org.postgresql.util.PSQLException: ERROR: co key value violates unique constraint "patch_set_approvals_pkey"
  Detail: Key (change_id, patch_set_id, account_id, category_id)=(6738979, 1, 1017701, Verified-Zoovy) already exists.

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:481)
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:840)
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1538)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at com.google.gwtorm.schema.sql.SqlDialect.executeBatch(SqlDialect.java:448)
at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:460)
at com.google.gwtorm.jdbc.JdbcAccess.insertAsBatch(JdbcAccess.java:215)
at com.google.gwtorm.jdbc.JdbcAccess.insert(JdbcAccess.java:166)
at com.google.gwtorm.jdbc.JdbcAccess.upsert(JdbcAccess.java:338)
at com.google.gerrit.server.ApprovalsUtil.addReviewers(ApprovalsUtil.java:255)
at com.google.gerrit.server.ApprovalsUtil.addReviewers(ApprovalsUtil.java:213)
at com.google.gerrit.server.change.PostReviewers$Op.updateChange(PostReviewers.java:413)
... 74 more

{code}

Reply all
Reply to author
Forward
0 new messages