problem with pick_open_reference_otus.py

36 views
Skip to first unread message

Tong Liu

unread,
Mar 17, 2016, 11:43:07 AM3/17/16
to Qiime 1 Forum
Hi!

When I run pick_open_reference_otus.py with my fna. file, the programm seems never will be finished.... my seqs.fna is only 39mb, and I got the out put of step1, step2, step3 and step4. but not pynast_aligned_seqs/rep_set_aligned.fasta.

Could anyone has suggestion about it?



Thanks!


Best Regards!


Tong

Embriette

unread,
Mar 17, 2016, 11:55:36 AM3/17/16
to Qiime 1 Forum
Hi Tong,

Are you running this command on your local machine? 
Can you also send the full command that you used, as well as your log file?

Thanks!

Embriette

Tong Liu

unread,
Mar 17, 2016, 12:29:03 PM3/17/16
to Qiime 1 Forum
Hi!

Thanks for your reply!

I am runnin this on Uppmax.

in the attachment are my log file of pick_open_reference_otus.py, full command and map file.

Since this is just a test for my samples, only two samples were selected.


Best Regards!


Tong
illumina.docx
map_tong_test.txt
log_20160317163138.txt

Embriette

unread,
Mar 17, 2016, 12:32:55 PM3/17/16
to Qiime 1 Forum
Hi Tong,

Thanks for providing all of your files! I can see from your log file that you have provided the full path for your reference file, but not for your input file or output directory. Can you specify full path lengths and try running the script again? Let me know how that works for you.

Embriette

Colin Brislawn

unread,
Mar 17, 2016, 1:33:43 PM3/17/16
to Qiime 1 Forum
Hello Tong,

I noticed in the final line of your log file, the following command was being run.
parallel_assign_taxonomy_uclust.py -i open_otu_pick/uclust_97/rep_set.fna -o open_otu_pick/uclust_97/uclust_assigned_taxonomy -T --jobs_to_start 16 

Each one of those jobs will index a fully copy of your database and hold it in memory. It possible that with 16x jobs, your computer runs out of RAM and everything slows to a crawl. 

You could try checking in on your available memory during this step to see if all your RAM is being used. (You can do this using the program 'top' or a system monitor.) 
You could also try uses fewer parts, say 4, so that you are less likely to run out of RAM.

Let me know if that helps!
Colin Brislawn

Tong Liu

unread,
Mar 18, 2016, 7:55:20 AM3/18/16
to Qiime 1 Forum

Hi!

 

Thanks for your advice, but I actually run it on Uppmax, a service in Uppsala, Sweden. the core I applied are 16 cores…..

 

 

Best regards!

 

 

 

 

 

 

 

Tong Liu

 

 

 

PhD-student

 

Department of Microbiology, SLU

 

Box 7025

 

SE-750 07 Uppsala

 

Sweden

 

Tel: +46 (0) 18 67 3359

Tong Liu

unread,
Mar 18, 2016, 7:56:49 AM3/18/16
to Qiime 1 Forum

Hi!

 

 

 

I write the full address this time, but the program still has the same problem.


Best Regards!


Tong

log_20160317181058.txt

Embriette

unread,
Mar 18, 2016, 11:34:12 AM3/18/16
to Qiime 1 Forum
Hi Tong,

I agree with Colin on this-what is probably happening is that you are running out of memory. Each of the 16 jobs you are trying to run is using the same amount of memory, so if each job is using 8GB of memory, the entire job is using 8 x 16, or 128GB of memory. Try reducing the number of parallel jobs you're running; for example, 4 as Colin suggested. Let us know how it works!

Embriette


Tong Liu

unread,
Mar 23, 2016, 11:42:57 AM3/23/16
to Qiime 1 Forum
Hi! Thanks for you suggestion!

the program still get stuck, but it seems stuck after step 4, so I am manually working through parallel_align_seqs_pynast.py now :)


Tong.

Best Regards!

Colin Brislawn

unread,
Mar 23, 2016, 1:20:14 PM3/23/16
to Qiime 1 Forum
I'm glad you got that step working.

The script parallel_align_seqs_pynast.py takes very little RAM, but lots of CPU time. You could try running it using all 16 jobs to make it go faster.

Let us know if you have any other questions,
Colin
Reply all
Reply to author
Forward
0 new messages