/bin/bash: .command.run: No such file or directory. When running Nextflow from inside a POD.

[Javier Corvi]

Hello,

I have a basic question:  I'm using k8s executor, running nextflow from inside a pod using "nextflow run pipeline.nf".   I'm getting the error

/bin/bash: .command.run: No such file or directory
I look around and realize that this error appears usually when the pod is not able to access the storage shared through the PVC.

There is a problem,  the creation of the PVC right now is ReadWriteOnce and not ReadWriteMany (as it is requested in the Nexflow Kubernetes documentation), because the k8s is not configured to work with ReadWriteMany PVC yet ...

Can this be the problem ?

Here is the full error:

executor >  k8s (1)
[52/dc7f8d] process > nlp_standard_preprocessing (1)              [100%] 1 of 1, failed: 1 ✘
[-        ] process > pretox_sentence_classifier                  -
[-        ] process > pretox_treatment_related_finding_annotation -
[-        ] process > sr_domain_field_identification              -
Error executing process > 'nlp_standard_preprocessing (1)'

Caused by:
  Process `nlp_standard_preprocessing (1)` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

  exec >> /workspace/base-execution/14-07-2022/pipeline.log
     echo "********************************************************************************************************************** "
     echo `date`
     echo "Start Pipeline Execution, Pipeline Version 2.3, workflow name: pretox_pipeline_14-07-2022_10_13_41"
     echo "Start nlp-standard-preprocessing"
     nlp-standard-preprocessing -i in -o /workspace/base-execution/14-07-2022/pretoxtm_internal_files/nlp_standard_preprocessing_output -a BSC -t 8 -f true
  echo "End nlp-standard-preprocessing"

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  /bin/bash: .command.run: No such file or directory

Work dir:
  /usr/src/app/work/52/dc7f8d0b55b36852684b17a8a23947

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

****

When I enter into the work dir I cannot find ".command.out" or even the ".command.err"

bash-4.4# cd /usr/src/app/work/52/dc7f8d0b55b36852684b17a8a23947
bash-4.4# cat .command.out
cat: .command.out: No such file or directory



I have a pod which uses the nextflow docker image and also has the kubectl installed to run.

The part of k8s in the config file is:

process {
   executor = 'k8s'
}

k8s {
  storageClaimName = 'pretoxtm-workflow-pvc'
  storageMountPath = '/workspace'
  storageSubPath = 'base-execution'
  namespace = 'kh'
  serviceAccount= 'xxxxxxxxx'
}

The definition of the deployment.yml in which I start the nextflow job is:

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   annotations:
     namespace: kh
   labels:
     app: pretoxtm-workflow
   name: pretoxtm-workflow
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: pretoxtm-workflow
   template:
     metadata:
       labels:
         app: pretoxtm-workflow
     spec:
       serviceAccountName: xxxxxxxxxx
       containers:
       - name: pretoxtm-wfapi
         image: xxxxxx:latest
         imagePullPolicy: Always
         ports:
         - containerPort: 8092
         volumeMounts:
           - mountPath: /workspace
             name: pretoxtm-workflow-data
       volumes:
         - name: pretoxtm-workflow-data
           persistentVolumeClaim:
             claimName: pretoxtm-workflow-pvc

[Javier Corvi]

Sorry now, I observed that whenI enter into the work dir I can find the files:

bash-4.4# cd /usr/src/app/work/fd/780688c1378929e665ed63187b45b5/
bash-4.4# ls -a

.  ..  .command.log  .command.run  .command.sh

And inside command.log indicates the same:

bin/bash: .command.run: No such file or directory