On Windows - files.nio.renameTo throws The process cannot access the file because it is being used..

[arch...@gmail.com]

Hi 

I am trying to run the examples\DecisionTreeRegression app. Once the ModelDataService is running, I try to load the data with the cli command 

"cdap cli call service DecisionTreeRegression.ModelDataService PUT labels body:file examples\DecisionTreeRegression\src\test\resources\sample_libsvm_data.txtg the 

I am getting the following error

2018-05-11 21:50:35,901 - DEBUG [ModelDataService-http-executor-68:c.c.c.e.d.ModelDataService$ModelHandler$1@110] - Unable to write file D:\ProgramFiles\cask\data\namespaces\default\data\trainingData\95c13762-7e73-4ccc-aa4c-65184ecd047d

org.apache.tephra.TransactionFailureException: Exception raised from TxRunnable.run() co.cask.cdap.internal.app.runtime.AbstractContext$2@2196fc17

at co.cask.cdap.data2.transaction.Transactions$CacheBasedTransactional.finishExecute(Transactions.java:237) ~[na:na]

at co.cask.cdap.data2.transaction.Transactions$CacheBasedTransactional.execute(Transactions.java:223) ~[na:na]

at co.cask.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:507) ~[na:na]

at co.cask.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:495) ~[na:na]

at co.cask.cdap.internal.app.runtime.service.http.BodyConsumerAdapter.finished(BodyConsumerAdapter.java:98) ~[na:na]

at co.cask.http.HttpMethodInfo.bodyConsumerFinish(HttpMethodInfo.java:149) [co.cask.http.netty-http-0.16.0.jar:na]

at co.cask.http.HttpMethodInfo.chunk(HttpMethodInfo.java:108) [co.cask.http.netty-http-0.16.0.jar:na]

at co.cask.http.HttpDispatcher.messageReceived(HttpDispatcher.java:40) [co.cask.http.netty-http-0.16.0.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) [io.netty.netty-3.6.6.Final.jar:na]

at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) [io.netty.netty-3.6.6.Final.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Caused by: java.nio.file.FileSystemException: D:\ProgramFiles\cask\data\namespaces\default\data\trainingData\95c13762-7e73-4ccc-aa4c-65184ecd047d -> D:\ProgramFiles\cask\data\namespaces\default\data\trainingData\labels: The process cannot access the file because it is being used by another process.

at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) ~[na:1.8.0_161]

at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[na:1.8.0_161]

at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) ~[na:1.8.0_161]

at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) ~[na:1.8.0_161]

at java.nio.file.Files.move(Files.java:1395) ~[na:1.8.0_161]

at org.apache.twill.filesystem.LocalLocation.renameTo(LocalLocation.java:233) ~[org.apache.twill.twill-common-0.12.1.jar:0.12.1]

at co.cask.cdap.data2.dataset2.lib.file.FileSetDataset$FileSetLocation.renameTo(FileSetDataset.java:472) ~[na:na]

at co.cask.cdap.data2.dataset2.lib.file.FileSetDataset$FileSetLocation.renameTo(FileSetDataset.java:472) ~[na:na]

at co.cask.cdap.examples.dtree.ModelDataService$ModelHandler$1.onFinish(ModelDataService.java:97) ~[unpacked/:na]

at co.cask.cdap.internal.app.runtime.service.http.BodyConsumerAdapter$1.run(BodyConsumerAdapter.java:101) ~[na:na]

at co.cask.cdap.internal.app.runtime.AbstractContext$2.run(AbstractContext.java:512) ~[na:na]

at co.cask.cdap.data2.transaction.Transactions$CacheBasedTransactional.finishExecute(Transactions.java:235) ~[na:na]

... 16 common frames omitted

Searching for this error lead me 

https://stackoverflow.com/questions/33924479/java-nio-file-filesystemexception-the-process-cannot-access-the-file-because-it

which states this is known issue on Windows and recommends use of Files.copy followed by delete rather than Files.move()

try {
        //Files.move(fileToBeInstalled.toPath(), installPathFile.toPath(), REPLACE_EXISTING);
        Files.copy(fileToBeInstalled.toPath(), installPathFile.toPath(), REPLACE_EXISTING);
        fileToBeInstalled.delete();
} catch (IOException ex) {
    MainClass.getMainClass(InstallerLinux.class.getName()).log(Level.SEVERE, null, ex);

}

Is there any way to overcome this issue without changing cdap code base ?

Regards

Chetana

[arch...@gmail.com]

Another reference : https://stackoverflow.com/questions/43609757/trouble-using-files-move-and-file-renameto-on-windows

[Terence Yim]

Hi,

I think it is a bug in the implementation of the ModelDataService example. Specifically in the `onFinish` method, it tries to rename the file before closing the output channel, which under Windows is not allowed (and using the copy + delete is not the right solution as there could be unflushed bytes in memory). Would you mind altering the order of the `renameTo` and `channel.close` and compile the example again? Specifically the method should look like this:

@Override
public void onFinish(HttpServiceResponder responder) throws Exception {
  // libsvm can't read multiple files, so always rename the location to 'labels'
  channel.close();
  location.renameTo(trainingData.getBaseLocation().append("labels"));
  responder.sendStatus(200);
}

I also attached the whole .java file for your convenience.

[arch...@gmail.com]

Thanks Terence, this solved the issue.

Regards

Chetana