java.io.FileNotFoundException happened on sge server

[Linjun Wu]

Hello everyone, i want to share one probleme i have com across these days when i run bpipe pipeline on sge server,
Randomly after some stage begins, the following error happened, could anyone help me out or give me some suggestions, the error information is listed below
java.io.FileNotFoundException: .bpipe/commandtmp/73/cmd_run.sh (Stale file handle)
        at bpipe.executor.SgeCommandExecutor.start(SgeCommandExecutor.groovy:133)
        at bpipe.executor.ThrottledDelegatingCommandExecutor.doStart(ThrottledDelegatingCommandExecutor.groovy:111)
        at bpipe.executor.ThrottledDelegatingCommandExecutor.start(ThrottledDelegatingCommandExecutor.groovy:71)
        at bpipe.CommandManager.start(CommandManager.groovy:199)
        at bpipe.PipelineContext.async(PipelineContext.groovy:1682)
        at bpipe.PipelineContext.async(PipelineContext.groovy)
        at bpipe.PipelineContext.exec(PipelineContext.groovy:1360)
        at bpipe.PipelineContext.exec(PipelineContext.groovy:1309)
        at bpipe.PipelineContext.invokeMethod(PipelineContext.groovy)
        at bpipe.PipelineDelegate.methodMissing(PipelineDelegate.groovy:156)
        at bpipe.PipelineDelegate.invokeMethod(PipelineDelegate.groovy)
        at stage_bpipe$_run_closure3.doCall(stage_bpipe.groovy:47)
        at bpipe.PipelineStage.invokeMethod(PipelineStage.groovy)
        at bpipe.PipelineStage$_runBody_closure4_closure18.doCall(PipelineStage.groovy:315)
        at bpipe.PipelineStage$_runBody_closure4_closure18.doCall(PipelineStage.groovy)
        at bpipe.ParamsBinding.withLocalVariables(Runner.groovy:996)
        at bpipe.PipelineStage$_runBody_closure4.doCall(PipelineStage.groovy:314)
        at bpipe.PipelineStage$_runBody_closure4.doCall(PipelineStage.groovy)
        at bpipe.PipelineStage.runBody(PipelineStage.groovy:302)
        at bpipe.PipelineStage.this$2$runBody(PipelineStage.groovy)
        at bpipe.PipelineStage.run(PipelineStage.groovy:221)
        at bpipe.Pipeline.runSegment(Pipeline.groovy:522)
        at bpipe.Pipeline$runSegment$13.call(Unknown Source)
        at bpipe.PipelineCategory$_splitOnMap_closure11_closure37.doCall(PipelineCategory.groovy:480)
        at bpipe.PipelineCategory$_splitOnMap_closure11_closure37.doCall(PipelineCategory.groovy)

however, if i run this task local(not on servers through sge), no error happens

[Chenhao Li]

Hi Linjun,

I have the same problem too. I think the problem is most likely caused by the NFS latency problem, such that the output file was not reflected on the master node. 

I recently tried snakemake and found that it had an option "--latency-wait" to solve the NFS problem. It had been suggested to add a sleep command to all your batch commands (e.g. sleep 60) by the developers earlier before that option was added. I haven't tried this workaround myself so I am not sure whether it will work for bpipe.

Regards,

Chenhao