Failed to open fts.tvs file for writing POPULATIONS

[Diushi Keri]

Dear Stacks community

I have a problem with populations. I am doing a first filter using denovo example as follow using differents values of M, n and m.

denovo_map.pl --samples /path/to/samples/  --popmap pop_map.txt -o /path/to/output --paired -m 3 -n 1 -M 1 -t -T 12 -X "populations: -r 0.8 --fstats --fasta_strict --vcf"

However, when populations module is working I have:

...

Calculating population-level summary statistics
  /usr/local/bin/populations -P /Users/diushi/stacks-2.1/denovo -M pop_map.txt -t 12  -r 0.8 --fstats --fasta_strict --vcf


denovo_map.pl: Aborted because the last command failed (1); see log file.

And in denovo.log:

Error: Failed to open '/Users/diushi/Downloads/stacks-2.1/denovo/populations.fst_X-Y.tsv' for writing.
Aborted.

denovo_map.pl: Aborted because the last command failed (1).

Do you know any solution?

Thank you for your help.

Diushi

[Julian Catchen]

Hi Diushi, presumably your output path is incorrect:

Error: Failed to open
'/Users/diushi/Downloads/stacks-2.1/denovo/populations.fst_X-Y.tsv' for
writing.

julian

Diushi Keri wrote on 9/10/18 2:43 AM:

> Dear Stacks community
>
> I have a problem with populations. I am doing a first filter using
> denovo example as follow using differents values of M, n and m.
>
> denovo_map.pl --samples /path/to/samples/  --popmap pop_map.txt -o
> /path/to/output --paired -m 3 -n 1 -M 1 -t -T 12 -X "populations: -r 0.8
> --fstats --fasta_strict --vcf"
>
> However, when populations module is working I have:
>

> *...

>
> Calculating population-level summary statistics
>   /usr/local/bin/populations -P /Users/diushi/stacks-2.1/denovo -M
> pop_map.txt -t 12  -r 0.8 --fstats --fasta_strict --vcf
>
>
> denovo_map.pl: Aborted because the last command failed (1); see log file.

> *
> *
> *
> And in denovo.log:
>
> *Error: Failed to open

> '/Users/diushi/Downloads/stacks-2.1/denovo/populations.fst_X-Y.tsv' for
> writing.
> Aborted.
>
> denovo_map.pl: Aborted because the last command failed (1).

> *

[Diushi Keri]

Thank you Julian, I appreciate your response.

I am working with 133 individuals (266 files-paired reads) and when I use only a part of the data set or the data that populations says that it can't open, populations works fine.

Do you know what is happened?

Thank you again for your time and help.

[Gary Longo]

Hi Diushi and Julian,

I think populations is crashing because you're maxing out the memory on your machine, especially since its running when your reduce the sample number being processed. I have a very large data set (~700 individuals with 42 populations) and run into the same issue where I get the error:

Error: Failed to open 'xxxxxxxxxx/populations.phistats_4-29.tsv' for writing.

Based on the error I removed the --fstats flag and populations is currently running properly. With 42 populations there are a lot of pairwise comparisons but I'm not sure if thats the cause for the huge memory footprint. To clarify, it fails almost immediately with the --fstats flag. I tried using --batch_size 1000 and even 100 with --fstats and both failed right away. From what I've read on forum it doesn't sound like this flag has been tested much? 

I'm running this on a machine with 500G of memory. Here is the error message from the log of this node.

Out of memory: Kill process 26051 (populations) score 993 or sacrifice child

Killed process 26051, UID 1137, (populations) total-vm:526568272kB, anon-rss:524519872kB, file-rss:748kB

Hope this helps.

Cheers,

Gary