run not terminated - samtools sort and index

[Sophie D]

Dear Jon and all,

I am running dDocent on 203 individuals with a reference genome, locally. I reinstalled Ubuntu (16.04) and dDocent (2.2.19) this month and I am afraid something isn't working anymore.
The dDocent run seems to be going very very slow after creating trim files for each individuals (last bam file created 18h ago). Trimming per individual seem to work successfully for all of them. It has created "bam.bai" for 168 individuals / 203. The general trim.log is empty and all bam.log files are empty. Nothing else has been created. The terminal says:

[E::bgzf_close] file write error
samtools sort: failed to create "STA32_019.bam": Input/output error
samtools index: failed to create or write index
samtools index: failed to create or write indexsamtools sort: failed to create

Thank you for help,

Sophie

[Jon Puritz]

Hi Sophie,

Given this is a new installation, I’m guessing that you need to increase your file limits (https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/).

Jon

-- 

Jon Puritz, PhD
Postdoctoral Research Associate
Northeastern University
Marine Science Center
430 Nahant Rd, Nahant, MA 01908

Webpage: MarineEvoEco.com

Email: 
jpu...@gmail.com 

Cell: 401-338-8739

"The most valuable of all talents is that of never using two words when one will do."
-Thomas Jefferson

--
You received this message because you are subscribed to the Google Groups "dDocent User Help Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ddocent+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sophie D]

Hi Jon,

thank you for your answer.

The following command displays the maximum number of open file descriptors:
cat /proc/sys/fs/file-max
6570229

That seems like a lot already, no? How much should that be?

I tried to increase the maximum number of open files as suggested in the page you sent me (but this number is lower than what I have now!):
sudo sysctl -w fs.file-max=100000

but I cannot edit /etc/sysctl.conf file:
I type "fs.file-max = 100000" and try to exit ":wq"

It says: "E45: 'readonly' option is set (add ! to override)"

So I did: "wq!"

It says: "WARNING: The file has been changed since reading it!!!
Do you really want to write to it (y/n)?y
"/etc/sysctl.conf" E212: Can't open file for writing
Press ENTER or type command to continue"

Any further, suggestions?

Sophie

[Sophie D]

Hi again,

Nevermind, I managed to modify the maximum number of open files.
I will try a new run even if the number of open files allowed seemed quite high before I edited the config file.

Any other suggestions?

Sophie

[Sophie D]

Hi again,

I noticed that when starting a run, I get the following non-usual message:

"dDocent 2.2.19

Contact jpu...@gmail.com with any problems

Checking for required software
/usr/local/bin/dDocent: line 83: [: : integer expression expected
/usr/local/bin/dDocent: line 89: [: : integer expression expected

All required software is installed!
dDocent run started Wed Jul 12 13:47:49 CEST 2017
203 individuals are detected. Is this correct? Enter yes or no and press [ENTER]"

Could that point to an issue?

Regards,

Sophie

[Sophie D]

Dear all,

I am running in similar problem after changing the number of maximum files open allowed. The dDocent_main.log says:
"Trimming reads
Using BWA to map reads.

samtools index: failed to create or write index"

Here is my dDocent.runs:
"Variables used in dDocent Run at Wed Jul 12 15:52:15 CEST 2017
Number of Processors
14
Maximum Memory
0
Trimming
yes
Assembly?
no
Type_of_Assembly

Clustering_Similarity%

Mapping_Reads?
yes
Mapping_Match_Value
1
Mapping_MisMatch_Value
4
Mapping_GapOpen_Penalty
6
Calling_SNPs?
yes
Email"

It looks like Trimmomatics trimmed all reads. Trim.log is empty. It created only a limited number of bam, bam.bai, indexed versions and bam.log 156/203. All the bam.log are empty.

Thank you for your suggestions,

Sophie