problem with pick_open_reference_otus.py

[Tong Liu]

Hi!

When I run pick_open_reference_otus.py with my fna. file, the programm seems never will be finished.... my seqs.fna is only 39mb, and I got the out put of step1, step2, step3 and step4. but not pynast_aligned_seqs/rep_set_aligned.fasta.

Could anyone has suggestion about it?

Thanks!

Best Regards!

Tong

[Embriette]

Hi Tong,

Are you running this command on your local machine? 

Can you also send the full command that you used, as well as your log file?

Thanks!

Embriette

[Tong Liu]

Hi!

Thanks for your reply!

I am runnin this on Uppmax.

in the attachment are my log file of pick_open_reference_otus.py, full command and map file.

Since this is just a test for my samples, only two samples were selected.

Best Regards!

Tong

[Embriette]

Hi Tong,

Thanks for providing all of your files! I can see from your log file that you have provided the full path for your reference file, but not for your input file or output directory. Can you specify full path lengths and try running the script again? Let me know how that works for you.

Embriette

[Colin Brislawn]

Hello Tong,

I noticed in the final line of your log file, the following command was being run.

parallel_assign_taxonomy_uclust.py -i open_otu_pick/uclust_97/rep_set.fna -o open_otu_pick/uclust_97/uclust_assigned_taxonomy -T --jobs_to_start 16 

Each one of those jobs will index a fully copy of your database and hold it in memory. It possible that with 16x jobs, your computer runs out of RAM and everything slows to a crawl. 

You could try checking in on your available memory during this step to see if all your RAM is being used. (You can do this using the program 'top' or a system monitor.) 

You could also try uses fewer parts, say 4, so that you are less likely to run out of RAM.

Let me know if that helps!

Colin Brislawn

[Tong Liu]

Hi!

 

Thanks for your advice, but I actually run it on Uppmax, a service in Uppsala, Sweden. the core I applied are 16 cores…..

 

 

Best regards!

 

 

 

 

 

 

 

Tong Liu

 

 

 

PhD-student

 

Department of Microbiology, SLU

 

Box 7025

 

SE-750 07 Uppsala

 

Sweden

 

Tel: +46 (0) 18 67 3359

[Tong Liu]

Hi!

 

 

 

I write the full address this time, but the program still has the same problem.

Best Regards!

Tong

[Embriette]

Hi Tong,

I agree with Colin on this-what is probably happening is that you are running out of memory. Each of the 16 jobs you are trying to run is using the same amount of memory, so if each job is using 8GB of memory, the entire job is using 8 x 16, or 128GB of memory. Try reducing the number of parallel jobs you're running; for example, 4 as Colin suggested. Let us know how it works!

Embriette

[Tong Liu]

Hi! Thanks for you suggestion!

the program still get stuck, but it seems stuck after step 4, so I am manually working through parallel_align_seqs_pynast.py now :)

Tong.

Best Regards!

[Colin Brislawn]

I'm glad you got that step working.

The script parallel_align_seqs_pynast.py takes very little RAM, but lots of CPU time. You could try running it using all 16 jobs to make it go faster.

Let us know if you have any other questions,
Colin