Restarting an older run

[Ilya Flyamer]

Hello,

Thank you for the fantastic tool!
I am starting to use `distiller` to analyse Hi-C data. The first time I am running it on real data I get sort: write failed: /local/442628.1.eddie/sort55vHYi: No space left on device for one of the jobs. I am assuming this is a problem with space for temporary files, so I wanted to change NXF_TEMP. I put it on top of my cluster.config and re-ran the pipeline with -resume option. But it started submitting the very first jobs instead of skipping them! I didn't want that and immediately aborted. I guess this happened because the environment for all processes changed and nextflow assumed it should re-run everything? So is there a way to force -resume and reuse the files I already have? If I simply revert the change it now compares to the most recent run (I guess) and still resubmits the very first jobs... Should I only add NXF_TEMP to the problematic processes now?

Best wishes,
Ilya

[Paolo Di Tommaso]

Hi Ilya, 

Thanks for your kind works. I'm replying quoting your message. 

On Wed, Nov 29, 2017 at 11:26 AM, Ilya Flyamer <fly...@gmail.com> wrote:

Hello,

I am starting to use `distiller` to analyse Hi-C data. The first time I am running it on real data I get sort: write failed: /local/442628.1.eddie/sort55vHYi: No space left on device for one of the jobs. I am assuming this is a problem with space for temporary files, so I wanted to change NXF_TEMP.

This look a problem specific with a tool that you are using in your pipeline, which tries to the local storage (let me guess, is it GATK?). Here, nextflow variables won't help here. You need to find the tool related setting. 

 

I put it on top of my cluster.config and re-ran the pipeline with -resume option. But it started submitting the very first jobs instead of skipping them! I didn't want that and immediately aborted. I guess this happened because the environment for all processes changed and nextflow assumed it should re-run everything?

Env variables generally does not have an impact on the resume mechanism, so it should have worked. I guess you have executed another (not resumed) run between the twos.

 

So is there a way to force -resume and reuse the files I already have? If I simply revert the change it now compares to the most recent run (I guess) and still resubmits the very first jobs... Should I only add NXF_TEMP to the problematic processes now?

You cannot force a resume, but you can list of all the runs you have launched and resume a specific one. Use the command `nextflow log` in the folder where your launching the pipeline. It provides a list of runs you launched. Fiend the one  you want to resume, then specify the run *name* or the session in as the resume option argument. For example: 

$ nextflow log

TIMESTAMP          DURATION RUN NAME        STATUS REVISION ID SESSION ID                          COMMAND                   

2017-11-24 18:41:34 672ms    ecstatic_noether OK    bab98280bf 7a8fefda-c812-4842-9248-2fd1b8d1d1e1 nextflow run <your pipeline>

2017-11-29 10:55:15 2.5s    grave_lavoisier OK    6a1acf3211 56c9a1a1-ad16-4671-b98a-96adbd5051f2 nextflow run <your pipeline>          

2017-11-29 10:55:29 -        golden_roentgen -      6a1acf3211 6b12ae11-74d8-4395-9685-4bb91e05e324 nextflow run <your pipeline>          

2017-11-29 09:57:37 6.2s    silly_austin    OK    6a1acf3211 a896b4da-4530-48e5-a519-39016adff6fb nextflow run <your pipeline>  

$ nextflow run <your pipeline> -resume grave_lavoisier   

Hope it helps. 

Cheers,
Paolo

[Ilya Flyamer]

Thanks a lot Paolo, this is great.

I did not execute anything else in between though... I did specify the project.yml file again though - should I not?

It is not GATK, it is https://github.com/mirnylab/distiller-nf, and the actual command was just a unix sort called inside one of the tools used in it, it seems.

Cheers,
ilya

среда, 29 ноября 2017 г., 13:37:05 UTC пользователь Paolo Di Tommaso написал:

[Ilya Flyamer]

Does restarting require only the work and the .nextflow folders?

среда, 29 ноября 2017 г., 14:16:30 UTC пользователь Ilya Flyamer написал:

[Paolo Di Tommaso]

Yes. 

Are you sure your pipeline is not changing some input files? just "touch"ing an input would change the file metadata an invalidate the tasks depending on it.

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Ilya Flyamer]

Thanks! Do you mean the original input files, not any intermediates? No, their modification date is not changed, at least.

I thought I might have done something wrong the previous time when restarting, but now I get the same error again... And I don't see it when starting the pipeline de novo. Any idea what it can be about, or is it a question for distiller devs? raw_reads_paths is definitely defined in project.yml

[s1529682@login04(eddie) distiller]$ nextflow run distiller.nf -resume sleepy_shockley
N E X T F L O W  ~  version 0.25.1
Launching `distiller.nf` [boring_hodgkin] - revision: fb683b24b8
WARN: Access to undefined parameter `input` -- Initialise it to a default value eg. `params.input = some_value`
ERROR ~ Cannot get property 'raw_reads_paths' on null object

 -- Check script 'distiller.nf' at line: 40 or see '.nextflow.log' file for more details

Thank you,
Ilya

четверг, 30 ноября 2017 г., 7:57:35 UTC пользователь Paolo Di Tommaso написал:

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

On Thu, Nov 30, 2017 at 10:59 AM, Ilya Flyamer <fly...@gmail.com> wrote:

Thanks! Do you mean the original input files, not any intermediates? No, their modification date is not changed, at least.

Any input files. Since you were mentioning it starts from the beginning I was supposing the original ones. 

 

I thought I might have done something wrong the previous time when restarting, but now I get the same error again... And I don't see it when starting the pipeline de novo. Any idea what it can be about, or is it a question for distiller devs? raw_reads_paths is definitely defined in project.yml

This looks a pipeline related problem. Sorry I can't help here. 

Cheers,
Paolo

 

[Ilya Flyamer]

OK, thanks a lot Paolo!

Best,
Ilya

четверг, 30 ноября 2017 г., 12:50:16 UTC пользователь Paolo Di Tommaso написал:

[Ilya Flyamer]

For the future, wanted to say that I had the problem with restarting because I didn't specify -params-file project.yml -profile cluster since I assumed that this information would be reused from the original command...

Ilya

четверг, 30 ноября 2017 г., 13:12:22 UTC пользователь Ilya Flyamer написал:

[Paolo Di Tommaso]

This sounds a nice idea to improve the resume mechanism !

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Ilya Flyamer]

Still, I again have a problem with restarting... I definitely use a correct command this time, but it starts from the very beginning of the pipeline. I don't know whether you could help me, but there is clearly something weird going on for me with restarting...

Ilya

пятница, 1 декабря 2017 г., 13:52:47 UTC пользователь Paolo Di Tommaso написал:

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

Are you are to replicate the problem? is it deterministic ? 

p 

[Ilya Flyamer]

Well, my data takes many hours to process, so I haven't tried yet... I will try it with a small test dataset included with distiller to check this.

вторник, 12 декабря 2017 г., 15:23:53 UTC пользователь Paolo Di Tommaso написал:

[Paolo Di Tommaso]

That's a very good idea. 

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.