Nexus - Unable to delete old blob repo_name@ ... for every deletions

[Frank Black]

Hi All,

I just noticed after deleting some assets / components using the GUI (and getting a successful message) that it was not deleting anything from the blob store itself. My set up is a bit different where the blob store is remote (UNC CIFS path). It works fine to push and pull components. There's no errors at all. I just noticed that it fails every time we delete a component though. Any thoughts? It talks about unknown network error, but connectivity is totally fine?

I'm using the latest version 3.6.1-02

2017-12-18 16:10:07,443-0800 WARN  [qtp2042251130-7011]  FLAPIERRE_P org.sonatype.nexus.repository.storage.BlobTx - Unable to delete old blob repo_name@E9333826-0C55AD6A-1EAD33C5-D7D6FDB5-93EC1EA0:b636b0ee-614a-406e-bb08-5f9643b9f8ca while committing transaction

org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: b636b0ee-614a-406e-bb08-5f9643b9f8ca, java.io.IOException: An unexpected network error occurred

at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:501)

at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)

at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)

at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)

at org.sonatype.nexus.repository.storage.BlobTx.commit(BlobTx.java:156)

at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:178)

at sun.reflect.GeneratedMethodAccessor672.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)

at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)

at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)

at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)

at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)

at com.sun.proxy.$Proxy221.commit(Unknown Source)

at org.sonatype.nexus.transaction.BatchTransaction.commit(BatchTransaction.java:38)

at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)

at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)

at org.sonatype.nexus.repository.maven.internal.MavenFacetUtils.deleteWithHashes(MavenFacetUtils.java:220)

at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUtils.delete(MetadataUtils.java:120)

at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.deleteAndRebuild(MetadataRebuilder.java:145)

at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.deleteMetadata(MavenHostedFacetImpl.java:122)

at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedComponentMaintenanceFacet.deleteComponent(MavenHostedComponentMaintenanceFacet.java:59)

at org.sonatype.nexus.repository.maintenance.internal.MaintenanceServiceImpl.deleteComponent(MaintenanceServiceImpl.java:92)

at org.sonatype.nexus.repository.maintenance.MaintenanceService$deleteComponent.call(Unknown Source)

at org.sonatype.nexus.coreui.ComponentComponent.deleteComponent(ComponentComponent.groovy:245)

at com.palominolabs.metrics.guice.ExceptionMeteredInterceptor.invoke(ExceptionMeteredInterceptor.java:49)

at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:47)

at org.sonatype.nexus.validation.internal.ValidationInterceptor.invoke(ValidationInterceptor.java:53)

at org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(AopAllianceMethodInvocationAdapter.java:49)

at org.apache.shiro.authz.aop.AuthorizingAnnotationMethodInterceptor.invoke(AuthorizingAnnotationMethodInterceptor.java:68)

at org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(AopAllianceMethodInterceptorAdapter.java:36)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.invokeJavaMethod(DispatcherBase.java:142)

at com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.invokeMethod(DispatcherBase.java:133)

at org.sonatype.nexus.extdirect.internal.ExtDirectServlet$3.invokeMethod(ExtDirectServlet.java:233)

at com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.dispatch(DispatcherBase.java:63)

at com.softwarementors.extjs.djn.router.processor.standard.StandardRequestProcessorBase.dispatchStandardMethod(StandardRequestProcessorBase.java:73)

at com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.processIndividualRequest(JsonRequestProcessor.java:502)

at com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.processIndividualRequestsInThisThread(JsonRequestProcessor.java:150)

at com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.process(JsonRequestProcessor.java:133)

at com.softwarementors.extjs.djn.router.RequestRouter.processJsonRequest(RequestRouter.java:83)

at com.softwarementors.extjs.djn.servlet.DirectJNgineServlet.processRequest(DirectJNgineServlet.java:617)

at com.softwarementors.extjs.djn.servlet.DirectJNgineServlet.doPost(DirectJNgineServlet.java:580)

at org.sonatype.nexus.extdirect.internal.ExtDirectServlet.doPost(ExtDirectServlet.java:138)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)

at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)

at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)

at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)

at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)

at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)

at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)

at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)

at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:102)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)

at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

at org.eclipse.jetty.server.Server.handle(Server.java:534)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

at java.lang.Thread.run(Unknown Source)

Caused by: java.io.IOException: An unexpected network error occurred

at java.io.RandomAccessFile.writeBytes(Native Method)

at java.io.RandomAccessFile.write(Unknown Source)

at com.squareup.tape.QueueFile.ringWrite(QueueFile.java:237)

at com.squareup.tape.QueueFile.add(QueueFile.java:314)

at com.squareup.tape.QueueFile.add(QueueFile.java:287)

at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:495)

... 114 common frames omitted

Thanks for your help,

[Rich Seddon]

It doesn't look like network connectivity is fine to me:

Caused by: java.io.IOException: An unexpected network error occurred

at java.io.RandomAccessFile.writeBytes(Native Method)

at java.io.RandomAccessFile.write(Unknown Source)

at com.squareup.tape.QueueFile.ringWrite(QueueFile.java:237)

at com.squareup.tape.QueueFile.add(QueueFile.java:314)

at com.squareup.tape.QueueFile.add(QueueFile.java:287)

at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:495)

... 114 common frames omitted

The stack is indicating some sort of network problem is occurring.  Note that this error is getting thrown from java.io.RandomAccessFile.writeBytes().  So Java is just trying to write content to a file, and the error is getting thrown.

[Frank Black]

Fair enough Rich. The problem I'm seeing now is that the web UI says "Success! Components deleted" with no errors at all, but in the logfile, it states that it failed with the network error. That leads me and my users to think that the pointers were deleted when they really weren't. The component doesn't show up anymore but the files on disk are still there...

[Frank Black]

To add on to my previous comment. After turning on DEBUG in the logger, I can see the asset gets deleted from the embedded database first (where the Success comes from in the UI?), and then further down you have the network failure. Shouldn't it be the other way round? Delete the file on disk first, then delete the pointer in the database? See below:

2017-12-27 14:38:55,684-0800 INFO  [qtp2022249924-712]  user org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedComponentMaintenanceFacet - Deleting asset: Asset{metadata=AttachedEntityMetadata{schema=asset, document=asset#29:3103{bucket:#23:10,format:maven2,last_updated:Tue Dec 19 12:19:51 PST 2017,attributes:[4],component:#27:887,name:component_name,size:9887,content_type:application/java-archive,created_by:nexus,created_by_ip::58176,blob_ref:hosted-test@E9333826-0C55AD6A-1EAD33C5-D7D6FDB5-93EC1EA0:f1eb9c9c-f1cf-401e-8c94-6f88162768f2,last_downloaded:Tue Dec 19 12:19:51 PST 2017,blob_created:Tue Dec 19 12:19:51 PST 2017,blob_updated:Tue Dec 19 12:19:51 PST 2017} v1}, name=component_name}

2017-12-27 14:38:55,684-0800 DEBUG [qtp2022249924-712]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: ACTIVE -> OPEN

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> ACTIVE

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: ACTIVE

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: ACTIVE -> OPEN

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: OPEN

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> CLOSED

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: CLOSED

2017-12-27 14:38:55,777-0800 WARN  [qtp2022249924-712]  user org.sonatype.nexus.repository.storage.BlobTx - Unable to delete old blob hosted-land@E9333826-0C55AD6A-1EAD33C5-D7D6FDB5-93EC1EA0:f1eb9c9c-f1cf-401e-8c94-6f88162768f2 while committing transaction

org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: f1eb9c9c-f1cf-401e-8c94-6f88162768f2, java.io.IOException: An unexpected network error occurred

        at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:501)

        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)

        at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)

        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)

        at org.sonatype.nexus.repository.storage.BlobTx.commit(BlobTx.java:156)

        at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:178)

.......

2017-12-27 14:38:55,777-0800 DEBUG [qtp2022249924-712]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: OPEN

2017-12-27 14:38:55,777-0800 DEBUG [qtp2022249924-712]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> CLOSED

I don't know why the network failure occurs, since I can easily put files (curl PUT) and there's no errors at all. Components can be accessed / downloaded. Seconds later, when I try to delete them, there's the network error showing up. Netstat shows a constant established session to the CIFS server. Looking at the firewall logs, all it shows is a previously established CIFS/445 session. If I restart the Nexus service, it works again.

Thoughts? I don't see anywhere that using UNC paths for the blob stores is unsupported. It's been working pretty well except for this issue.

Thanks,

Francois

[Peter Lynch]

On Wed, Dec 27, 2017 at 9:10 PM, Frank Black <undec...@gmail.com> wrote:

To add on to my previous comment. After turning on DEBUG in the logger, I can see the asset gets deleted from the embedded database first (where the Success comes from in the UI?), and then further down you have the network failure. Shouldn't it be the other way round? Delete the file on disk first, then delete the pointer in the database? See below:

2017-12-27 14:38:55,684-0800 INFO  [qtp2022249924-712]  user org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedComponentMaintenanceFacet - Deleting asset: Asset{metadata=AttachedEntityMetadata{schema=asset, document=asset#29:3103{bucket:#23:10,format:maven2,last_updated:Tue Dec 19 12:19:51 PST 2017,attributes:[4],component:#27:887,name:component_name,size:9887,content_type:application/java-archive,created_by:nexus,created_by_ip::58176,blob_ref:hosted-test@E9333826-0C55AD6A-1EAD33C5-D7D6FDB5-93EC1EA0:f1eb9c9c-f1cf-401e-8c94-6f88162768f2,last_downloaded:Tue Dec 19 12:19:51 PST 2017,blob_created:Tue Dec 19 12:19:51 PST 2017,blob_updated:Tue Dec 19 12:19:51 PST 2017} v1}, name=component_name}

2017-12-27 14:38:55,684-0800 DEBUG [qtp2022249924-712]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: ACTIVE -> OPEN

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> ACTIVE

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: ACTIVE

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: ACTIVE -> OPEN

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: OPEN

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> CLOSED

2017-12-27 14:38:55,699-0800 DEBUG [event-7-thread-198]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: CLOSED

2017-12-27 14:38:55,777-0800 WARN  [qtp2022249924-712]  user org.sonatype.nexus.repository.storage.BlobTx - Unable to delete old blob hosted-land@E9333826-0C55AD6A-1EAD33C5-D7D6FDB5-93EC1EA0:f1eb9c9c-f1cf-401e-8c94-6f88162768f2 while committing transaction

org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: f1eb9c9c-f1cf-401e-8c94-6f88162768f2, java.io.IOException: An unexpected network error occurred

        at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:501)

        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)

        at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)

        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)

        at org.sonatype.nexus.repository.storage.BlobTx.commit(BlobTx.java:156)

        at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:178)

.......

2017-12-27 14:38:55,777-0800 DEBUG [qtp2022249924-712]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: OPEN

2017-12-27 14:38:55,777-0800 DEBUG [qtp2022249924-712]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> CLOSED

I don't know why the network failure occurs, since I can easily put files (curl PUT) and there's no errors at all. Components can be accessed / downloaded. Seconds later, when I try to delete them, there's the network error showing up. Netstat shows a constant established session to the CIFS server. Looking at the firewall logs, all it shows is a previously established CIFS/445 session. If I restart the Nexus service, it works again.

Thoughts? I don't see anywhere that using UNC paths for the blob stores is unsupported. It's been working pretty well except for this issue.

The area of stack trace we need to focus on is this:

Caused by: java.io.IOException: An unexpected network error occurred

at java.io.RandomAccessFile.writeBytes(Native Method)

at java.io.RandomAccessFile.write(Unknown Source)

at com.squareup.tape.QueueFile.ringWrite(QueueFile.java:237)

at com.squareup.tape.QueueFile.add(QueueFile.java:314)

at com.squareup.tape.QueueFile.add(QueueFile.java:287)

at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:495)

The message "An unexpected network error occurred" is a native operating system message from Windows. The "bug" resides someplace outside of Nexus code in either:

- the "Native Method" code that is specific to writing bytes to a random access file

- the specific version of the JVM that you are using

- the specific operating system version or patched version that Nexus is running on

- some operating system specific setting for your network share

- an underlying or general network problem

What I would do first is check if you can reproduce this using the latest patch level of operating system and the Oracle JDK8 available for it.

 

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/a4e53dbf-d2e3-4272-b087-32a7b15fb1b9%40glists.sonatype.com.

For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Frank Black]

I can reproduce it at any point on that specific repository.

Here, I'm pushing a test project successfully, then seconds later, I try to delete it and it says success in the UI, but fails.

###############

Request.log

###############

IP:49979 - user [28/Dec/2017:11:07:11 -0800] "PUT /repository/maven-test-snapshots/test/test/my-app/1.0-SNAPSHOT/my-app-1.0-SNAPSHOT.jar HTTP/1.1" 201 0 313 "curl/7.35.0"

###############

Nexus.log

###############

2017-12-28 11:07:11,550-0800 DEBUG [qtp2022249924-711]  user org.sonatype.nexus.repository.storage.DefaultContentValidator - Content test/test/my-app/1.0-SNAPSHOT/my-app-1.0-SNAPSHOT.jar declared as null, determined as application/java-archive

2017-12-28 11:07:11,550-0800 DEBUG [qtp2022249924-711]  user org.sonatype.nexus.repository.storage.ComponentEntityAdapter - Finding components with query: select from component where (group = :p0 AND name = :p1 AND version = :p2) and (bucket=#23:10), parameters: {p0=test.test, p1=my-app, p2=1.0-SNAPSHOT}

2017-12-28 11:07:11,628-0800 DEBUG [qtp2022249924-711]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: ACTIVE -> OPEN

2017-12-28 11:07:11,628-0800 DEBUG [qtp2022249924-711]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: OPEN

2017-12-28 11:07:11,628-0800 DEBUG [qtp2022249924-711]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> CLOSED

2017-12-28 11:07:11,628-0800 DEBUG [qtp2022249924-711]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: CLOSED

......

2017-12-28 11:07:36,909-0800 DEBUG [qtp2022249924-54]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: CLOSED

2017-12-28 11:07:36,909-0800 DEBUG [qtp2022249924-54]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> ACTIVE

2017-12-28 11:07:36,909-0800 DEBUG [qtp2022249924-54]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: ACTIVE

2017-12-28 11:07:36,909-0800 INFO  [qtp2022249924-54]  user org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedComponentMaintenanceFacet - Deleting component: Component{metadata=AttachedEntityMetadata{schema=component, document=component#28:1593{bucket:#23:10,format:maven2,last_updated:Thu Dec 28 11:07:11 PST 2017,attributes:[1],group:test.test,name:my-app,version:1.0-SNAPSHOT,ci_name:my-app} v1}, name=my-app, version=1.0-SNAPSHOT, group=test.test}

2017-12-28 11:07:36,909-0800 DEBUG [qtp2022249924-54]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: ACTIVE -> OPEN

2017-12-28 11:07:36,987-0800 WARN  [qtp2022249924-54]  user org.sonatype.nexus.repository.storage.BlobTx - Unable to delete old blob hosted-test@E9333826-0C55AD6A-1EAD33C5-D7D6FDB5-93EC1EA0:53463fdf-f3b2-4d29-9d15-74c96ff7268c while committing transaction

org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: 53463fdf-f3b2-4d29-9d15-74c96ff7268c, java.io.IOException: An unexpected network error occurred

        at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:501)

        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)

        at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)

        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)

        at org.sonatype.nexus.repository.storage.BlobTx.commit(BlobTx.java:156)

        at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:178)

        at sun.reflect.GeneratedMethodAccessor256.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

        at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)

        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)

        at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)

        at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)

        at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)

        at com.sun.proxy.$Proxy230.commit(Unknown Source)

        at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)

        at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)

        at org.sonatype.nexus.repository.storage.DefaultComponentMaintenanceImpl.deleteComponent(DefaultComponentMaintenanceImpl.java:45)

        at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedComponentMaintenanceFacet.deleteComponent(MavenHostedComponentMaintenanceFacet.java:56)

        ... 110 common frames omitted

2017-12-28 11:07:36,987-0800 DEBUG [qtp2022249924-54]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: OPEN

2017-12-28 11:07:36,987-0800 DEBUG [qtp2022249924-54]  user org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> CLOSED

Thanks,

[Frank Black]

Thanks Peter. I'm using the latest Oracle JRE. I'll upgrade to 3.7.1 and see if I can reproduce. If so, I'll switch back to the embedded JRE and see if I can reproduce.

Can you comment on the fact that it deletes the entry in the database before confirming successful deletion of the file on disk? Now, I'm more concerned about the "success" deletion from the database while the files don't get deleted from disk.

Thanks,

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.

[Peter Lynch]

On Thu, Dec 28, 2017 at 3:49 PM, Frank Black <undec...@gmail.com> wrote:

Thanks Peter. I'm using the latest Oracle JRE. I'll upgrade to 3.7.1 and see if I can reproduce. If so, I'll switch back to the embedded JRE and see if I can reproduce.

To clarify, my suggestions are not about upgrading Nexus, more to do with using latest operating system patch levels and Java version since we are talking about native code. You will have to override what version of Java Nexus is running with per https://help.sonatype.com/display/NXRM3/Java+Runtime+Environment

Can you comment on the fact that it deletes the entry in the database before confirming successful deletion of the file on disk? Now, I'm more concerned about the "success" deletion from the database while the files don't get deleted from disk.

I suggest this specific concern be filed in the NEXUS project at https://issues.sonatype.org and cite this mailing list thread.

 

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/d97692a4-83e0-4e55-b57e-0b3be48eb9d5%40glists.sonatype.com.

[Frank Black]

Created JIRA ticket 15358 to track the deletion sequence issue: https://issues.sonatype.org/browse/NEXUS-15358

Thanks for the clarification. I understand and have been overriding the default JRE on windows by creating a pref_jre.cfg in the .install4j directory pointing to the system JRE. I'll update that JRE (if a new version is available) and also test by switching back to the embedded JRE.