Unable to run tests on spark scala maven archetype project

[Vamsi Sudhir]

Hello,

I have setup a project using cdap scala archetype and imported the project in intellij on windows.

When i try to run the cdap unit test(SparkkMeansAppTest) i see the below exception.

I have tried changing the temp folder permissions. i see that the test is run with the current user on windows and manually i am able to delete files/folder under this location.

Could you please check this.

java.io.IOException: Failed to delete file C:\Users\u0122207\AppData\Local\Temp\junit2577230884642326546\junit3670540051017568513\tmp\spark.default.SparkKMeans.SparkKMeansProgram.31b4ce42-99fc-11e6-996b-00000098b99c\unpacked\lib\core-1.1.2.jar

at co.cask.cdap.common.utils.DirUtils.deleteDirectoryContents(DirUtils.java:75) ~[cdap-common-3.5.1.jar:na]

at co.cask.cdap.common.utils.DirUtils.deleteDirectoryContents(DirUtils.java:49) ~[cdap-common-3.5.1.jar:na]

at co.cask.cdap.app.runtime.AbstractProgramRuntimeService$1.run(AbstractProgramRuntimeService.java:183) [cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.app.runtime.AbstractProgramRuntimeService$1.run(AbstractProgramRuntimeService.java:190) [cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.app.runtime.AbstractProgramRuntimeService.remove(AbstractProgramRuntimeService.java:442) [cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.app.runtime.AbstractProgramRuntimeService.access$300(AbstractProgramRuntimeService.java:82) [cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.app.runtime.AbstractProgramRuntimeService$4.completed(AbstractProgramRuntimeService.java:407) [cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.internal.app.runtime.AbstractProgramController$ListenerCaller$7.run(AbstractProgramController.java:481) [cdap-app-fabric-3.5.1.jar:na]

at org.apache.twill.common.Threads$1.execute(Threads.java:35) [twill-common-0.7.0-incubating.jar:0.7.0-incubating]

at co.cask.cdap.internal.app.runtime.AbstractProgramController$ListenerCaller.completed(AbstractProgramController.java:478) [cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.internal.app.runtime.AbstractProgramController$MultiListenerCaller.completed(AbstractProgramController.java:384) [cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.internal.app.runtime.AbstractProgramController$5.run(AbstractProgramController.java:201) [cdap-app-fabric-3.5.1.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

2016-10-24 20:42:03,018 - WARN  [main:c.c.c.i.a.n.DefaultNamespaceAdmin@291] - Error while deleting namespace namespace:default

java.lang.RuntimeException: java.nio.file.AccessDeniedException: C:\Users\u0122207\AppData\Local\Temp\junit2577230884642326546\junit3670540051017568513\namespaces\default\streams\pointsStream -> C:\Users\u0122207\AppData\Local\Temp\junit2577230884642326546\junit3670540051017568513\namespaces\default\streams\.deleted\pointsStream1477321923016

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-13.0.1.jar:na]

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin$16.run(FileStreamAdmin.java:675) ~[cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data.stream.AbstractStreamCoordinatorClient.deleteStream(AbstractStreamCoordinatorClient.java:121) ~[cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin.doDrop(FileStreamAdmin.java:626) ~[cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin.dropAllInNamespace(FileStreamAdmin.java:189) ~[cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.internal.app.namespace.DefaultNamespaceResourceDeleter.deleteResources(DefaultNamespaceResourceDeleter.java:101) ~[cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin.delete(DefaultNamespaceAdmin.java:271) ~[cdap-app-fabric-3.5.1.jar:na]

at co.cask.cdap.test.TestBase.finish(TestBase.java:445) [cdap-unit-test-3.5.1.jar:na]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_65]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_65]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_65]

at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65]

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) [junit-4.11.jar:na]

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.11.jar:na]

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) [junit-4.11.jar:na]

at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) [junit-4.11.jar:na]

at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.11.jar:na]

at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.11.jar:na]

at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.11.jar:na]

at org.junit.runners.ParentRunner.run(ParentRunner.java:309) [junit-4.11.jar:na]

at co.cask.cdap.common.test.TestRunner.run(TestRunner.java:68) [cdap-common-3.5.1-tests.jar:na]

at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) [surefire-junit4-2.14.1.jar:2.14.1]

at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) [surefire-junit4-2.14.1.jar:2.14.1]

at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) [surefire-junit4-2.14.1.jar:2.14.1]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_65]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_65]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_65]

at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65]

at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208) [surefire-api-2.14.1.jar:2.14.1]

at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159) [surefire-booter-2.14.1.jar:2.14.1]

at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87) [surefire-booter-2.14.1.jar:2.14.1]

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) [surefire-booter-2.14.1.jar:2.14.1]

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95) [surefire-booter-2.14.1.jar:2.14.1]

Caused by: java.nio.file.AccessDeniedException: C:\Users\u0122207\AppData\Local\Temp\junit2577230884642326546\junit3670540051017568513\namespaces\default\streams\pointsStream -> C:\Users\u0122207\AppData\Local\Temp\junit2577230884642326546\junit3670540051017568513\namespaces\default\streams\.deleted\pointsStream1477321923016

at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) ~[na:1.8.0_65]

at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[na:1.8.0_65]

at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) ~[na:1.8.0_65]

at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) ~[na:1.8.0_65]

at java.nio.file.Files.move(Files.java:1395) ~[na:1.8.0_65]

at org.apache.twill.filesystem.LocalLocation.renameTo(LocalLocation.java:178) ~[twill-common-0.7.0-incubating.jar:0.7.0-incubating]

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin$16$2.call(FileStreamAdmin.java:664) ~[cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin$16$2.call(FileStreamAdmin.java:651) ~[cdap-data-fabric-3.5.1.jar:na]

at co.cask.cdap.common.security.ImpersonationUtils$1.run(ImpersonationUtils.java:46) ~[cdap-common-3.5.1.jar:na]

at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_65]

at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_65]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) ~[hadoop-common-2.3.0.jar:na]

at co.cask.cdap.common.security.ImpersonationUtils.doAs(ImpersonationUtils.java:43) ~[cdap-common-3.5.1.jar:na]

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin$16.run(FileStreamAdmin.java:651) ~[cdap-data-fabric-3.5.1.jar:na]

... 31 common frames omitted

Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 34.331 sec <<< FAILURE!

com.ecp.ingest.bulk.SparkKMeansAppTest  Time elapsed: 29.544 sec  <<< ERROR!

co.cask.cdap.common.NamespaceCannotBeDeletedException: 'namespace:default' could not be deleted. Reason: java.nio.file.AccessDeniedException: C:\Users\u0122207\AppData\Local\Temp\junit2577230884642326546\junit3670540051017568513\namespaces\default\streams\pointsStream -> C:\Users\u0122207\AppData\Local\Temp\junit2577230884642326546\junit3670540051017568513\namespaces\default\streams\.deleted\pointsStream1477321923016

at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)

at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)

at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)

at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)

at java.nio.file.Files.move(Files.java:1395)

at org.apache.twill.filesystem.LocalLocation.renameTo(LocalLocation.java:178)

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin$16$2.call(FileStreamAdmin.java:664)

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin$16$2.call(FileStreamAdmin.java:651)

at co.cask.cdap.common.security.ImpersonationUtils$1.run(ImpersonationUtils.java:46)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

at co.cask.cdap.common.security.ImpersonationUtils.doAs(ImpersonationUtils.java:43)

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin$16.run(FileStreamAdmin.java:651)

at co.cask.cdap.data.stream.AbstractStreamCoordinatorClient.deleteStream(AbstractStreamCoordinatorClient.java:121)

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin.doDrop(FileStreamAdmin.java:626)

at co.cask.cdap.data2.transaction.stream.FileStreamAdmin.dropAllInNamespace(FileStreamAdmin.java:189)

at co.cask.cdap.internal.app.namespace.DefaultNamespaceResourceDeleter.deleteResources(DefaultNamespaceResourceDeleter.java:101)

at co.cask.cdap.internal.app.namespace.DefaultNamespaceAdmin.delete(DefaultNamespaceAdmin.java:271)

at co.cask.cdap.test.TestBase.finish(TestBase.java:445)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)

at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)

at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)

at org.junit.rules.RunRules.evaluate(RunRules.java:20)

at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

at co.cask.cdap.common.test.TestRunner.run(TestRunner.java:68)

at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)

at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)

at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)

at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159)

at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87)

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

[Ali Anwar]

Hi.

This is an issue which was previously found, but not fixed for all cases, so I've reopened the JIRA: https://issues.cask.co/browse/CDAP-4695.

In the meantime - as a workaround - can you use the VirtualMachine on Windows to execute unit tests?

Regards,

Ali Anwar

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/cf9af6e6-b29f-4562-8337-9b3675d4116a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.