simpler identification of nextflow jobs in SGE queue with qstat

[Fabien Campagne]

We now use Nextflow with SGE on the cloud environment configured with NextflowWorkbench. We can start jobs from the workbench, and they are submitted to SGE as expected.

When looking at the output of qstat (used to monitor the progress of SGE jobs), we see:

ubuntu@frontend001:~$ qstat

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 

-----------------------------------------------------------------------------------------------------------------

    379 0.75000 nf-Kallist ubuntu       r     04/02/2016 13:01:58 all.q@compute002                   1        

    380 0.58696 nf-Kallist ubuntu       r     04/02/2016 13:24:58 all.q@compute003                   1        

    381 0.58696 nf-Kallist ubuntu       r     04/02/2016 13:27:28 all.q@compute005                   1        

    382 0.58696 nf-Kallist ubuntu       r     04/02/2016 13:49:13 all.q@compute004                   1        

    383 0.58696 nf-Kallist ubuntu       r     04/02/2016 13:54:28 all.q@compute001                   1        

    384 0.58696 nf-Kallist ubuntu       qw    04/02/2016 05:19:28                                    1        

    385 0.42391 nf-Kallist ubuntu       qw    04/02/2016 05:19:43                                    1        

    386 0.42391 nf-Kallist ubuntu       qw    04/02/2016 05:19:43                                    1        

    387 0.42391 nf-Kallist ubuntu       qw    04/02/2016 05:19:43                                    1        

    388 0.41304 nf-Kallist ubuntu       qw    04/02/2016 05:19:44                                    1        

    389 0.41304 nf-Kallist ubuntu       qw    04/02/2016 05:19:44                                    1        

    390 0.25000 nf-Kallist ubuntu       qw    04/02/2016 05:19:59                                    1    

As you can see, the job name is set to a long string that starts with a constant bit: nf-Kallist. qstat shows only the beginning of the job name (first 10 chars). 

We can use qstat -j to find out the full name, for instance:

job_name: nf-KallistoHDF5WithTuples_(32)

When starting this process, nextflow printed:

  [sshexec] [d1/d1bc6c] Submitted process > KallistoHDF5WithTuples (32)

So I think it would be nice to set the beginning of the job name to something that is more informative when looking at a qstat output. For instance, I think it would be useful to either

1. set the name to the start of the d1/d1bc6c string.

2. set the name to start with 32|d1/d1bc6c 

Paolo, is there a simple way to configure this with nextflow, or is something that could be added? 

Thanks. Fabien

[Paolo Di Tommaso]

Hi  Fabien, 

Congrats for your integration of NextflowWorkbench with Google cloud. It looks neat. 

Regarding your request, in the past we discussed about the job names to be displayed by the grid scheduler and at the end we decided to not use the process hash number because it would not be easily human readable. 

However I understand users may have different requirements depending specific use cases, thus a solution could be to add a directive that could be used to customize the process name or the cluster job name as you prefer. For example: 

  process foo {

    jobName "${ <your logic here> }"

  

    :

   } 

Do you think this could work for you? 

Best,
Paolo

If you think this could wor 

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Fabien Campagne]

Hi Paolo,

Yes, I think this would be great (for some reason I was not notified you posted an answer and just saw your response). 

Fabien

[Paolo Di Tommaso]

Hi Fabien, 

Could you please open a feature request on GitHub for this? 

Thanks,

Paolo

[Fabien Campagne]

Sure. Done.