Killed 9 error message in generating genomes when I am doing everything I thought I should

[Richard Friedman]

Dear Alex and STAR community,

I am new to STAR. I wish to install and index the human genome on
a mac with 32 Gb or RAM in 4 slots. I am using STAR 2.4.2a (Mac executable).
I have Mac OX 10.10.3 . I used the following command:

c2b2afmd2:stardir friedman$ ./STAR --runThreadN 1  --runMode genomeGenerate --genomeDir /Applications/stardir/GRCh38genomedir/ --genomeFastaFiles /Applications/stardir/GRCh38fasta/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna --sjdbGTFfile /Applications/stardir/gencode.v20.annotation.gtf —-genomeChrBinNbits 12 --limitGenomeGenerateRAM 24000000000
Jul 07 16:19:43 ..... Started STAR run
Jul 07 16:19:43 ... Starting to generate Genome files
Jul 07 16:20:34 ... starting to sort  Suffix Array. This may take a long time...
Jul 07 16:20:47 ... sorting Suffix Array chunks and saving them to disk...
Jul 07 17:39:09 ... loading chunks from disk, packing SA...
Jul 07 17:41:14 ... Finished generating suffix array
Jul 07 17:41:14 ... starting to generate Suffix Array index...
Jul 07 17:57:50 ..... Processing annotations GTF
Jul 07 17:58:12 ..... Inserting junctions into the genome indices
Killed: 9

When I had tried it without the annotation file and I tried to do the alignment it ran over the weekend without aligning.
Please advise.

Thanks and best wishes,
Rich
Richard A, Friedman, PhD
Herbert Irving Comprehensive Cancer Center
Columbia University

[Alexander Dobin]

Hi Rich,

the "Killed 9" problem is most likely related to RAM. I was trying to generate this genome on a 32GB server, and it did not work. It looks like some changes in 2.4.1 increase the required RAM just above 32GB (to ~33-34). I will release a patch shortly.

For the the other problem, please send me the Log.out files for the genome generation step and mapping.

Cheers

Alex

[Richard Friedman]

Alex,

Thank you for your  reply and your offer to patch the program.

I am not sure what I did over the weekend but when I subsequently ran the genome index generation and
alignment without GTF input  ran until it seemed to have generated a SAM file except that it did not run to completion.
I am attaching that Log.out session so that you can see how far it ran and perhaps tell me if
the sam file it generated is good. Also, if you can keep it at 31Gb it will help it run on a mac
because the system etc takes up some memory. But I guess that you have factored that in to your
work previously.

A few more questions:

1. A program called STARlong is part of the Mac distribution.
When if ever should I use it?
2. Is there something to be gained from compiling from source
rather than the precompiled version.
3. I have a 32Gb Mac consisting of quad core intel 3.5 i5 with 8Gb memory in each 4 slots.
Is it less likely to get a killed 9 if I run -runThreadN 4 or -runThreadN 1.

Thanks and best wishes,
Rich

[Alexander Dobin]

Hi Rich,

1. I recommend switching to STARlong for reads longer than 200-300 bases.

2. Compiling from source, in principle, might give you a slightly better performance, or resolve some kinds of OS incompatibility issues.

If the pre-compiled binaries work fine, there is no need to compile from the source.

3. For the genome generation step, the amount of RAM used does not depend on the number of threads. 

For mapping, STAR needs 150MB per thread for buffering. 32 GB should be enough for mapping with 4 threads.

You can download this genome index and try to map to it to see whether it works.

Cheers

Alex

[Richard Friedman]

Alex,

I downloaded the files in the directory for which you gave me the link
and I ran the following command:

c2b2afmd2:stardir friedman$ ./STAR --runThreadN 4 --runMode alignReads --genomeDir /Applications/stardir/GRCh38genomedir/ --readFilesIn /Applications/stardir/retina_gse40524/SRR548611/SRR548611_1.fastq /Applications/stardir/retina_gse40524/SRR548611/SRR548611_2.fastq --outFileNamePrefix SRR548611-GRCh38 --outSAMtype BAM SortedByCoordinate --quantMode GeneCounts

AND GOT THE FOLLOWING OUTPUT AND ERROR MESSAGE:

Jul 13 16:34:15 ..... Started STAR run
Jul 13 16:34:15 ..... Loading genome

ERROR_011003: exiting because of *INPUT FILE* error: could not open input file /Applications/stardir/GRCh38genomedir//exonGeTrInfo.tab
Solution: check that the file exists and you have read permission for this file
          SOLUTION: utilize --sjdbGTFfile /path/to/annotantions.gtf option at the genome generation step or mapping step

Jul 13 16:36:38 ...... FATAL ERROR, exiting

SO I RAN IT AGAIN WITH THE GTF FILE


c2b2afmd2:stardir friedman$ ./STAR --runThreadN 4 --runMode alignReads --genomeDir /Applications/stardir/GRCh38genomedir/ --readFilesIn /Applications/stardir/retina_gse40524/SRR548611/SRR548611_1.fastq /Applications/stardir/retina_gse40524/SRR548611/SRR548611_2.fastq --sjdbGTFfile /Applications/stardir/gencode.v21.annotation.gtf --outFileNamePrefix SRR548611-GRCh38 --outSAMtype BAM SortedByCoordinate --quantMode GeneCounts
Jul 13 16:40:38 ..... Started STAR run
Jul 13 16:40:38 ..... Loading genome

EXITING because of FATAL ERROR: old Genome is INCOMPATIBLE with on the fly junction insertion
SOLUTION: please re-generate genome from scratch with the latest version of STAR

BUY I CANNOT RE-GENERATE THE GENOME DUE TO THE PROGRAM EXITING WHEN
IT PASSES 32 Gb, AS DISCUSSED EARLIER.

THANKS AND BEST WISHES,
Rich

[Alexander Dobin]

Hi Rich,

this genome was generated with the old version of STAR, which did not support the new --quantMode GeneCounts option.

You can try to map with this option just to see if mapping works on your machine.

I am re-generating several genomes with the latest STAR, and one of them is available:

http://labshare.cshl.edu/shares/gingeraslab/www-data/dobin/STAR/STARgenomes/GENCODE/GRCh38_Gencode22/

With this one, you can use the --quantMode GeneCounts option.

Sorry about the confusion with the versions.

Cheers

Alex

[Richard Friedman]

Alex,

I ran it with and without sorting a bam file and it bombed both times.
I am attaching the log file from the run without the bam file.

Thanks and best wishes,
Rich

[Alexander Dobin]

Hi Rich,

was it killed with "Killed 9" message?

The loading of the genome went through OK.

Please check that you do not have user RAM limited by ulimit.

Also, please try the newly re-generated genome:

http://labshare.cshl.edu/shares/gingeraslab/www-data/dobin/STAR/STARgenomes/GENCODE/GRCh38_Gencode22/

Cheers

Alex

[Richard Friedman]

Alex,

Thanks.

On Tuesday, July 14, 2015 at 6:21:27 PM UTC-4, Alexander Dobin wrote:

Hi Rich,

was it killed with "Killed 9" message?

Yes:
Jul 14 14:28:11 ..... Started mapping
Killed: 9
 

 The loading of the genome went through OK.

Please check that you do not have user RAM limited by ulimit.

c2b2afmd2:stardir friedman$ ulimit
unlimited
 

Also, please try the newly re-generated genome:

http://labshare.cshl.edu/shares/gingeraslab/www-data/dobin/STAR/STARgenomes/GENCODE/GRCh38_Gencode22/

will do!

Best wishes,
Rich

[Richard Friedman]

Alex,

I tried it with the new genome indices both with and without making BAM counts based on the gtf file.
My log files for both sessions are attached.

The second run, without sorting  produced a 982 Mb sam file before bombing.
Is it possible to tell from the log file if the sam file consists of a complete
alignment? If so, I can compress it to bam, and count the reads, and perhaps order it
with another tool. Please advise.


Here is a record of each session:

c2b2afmd2:stardir friedman$ ./STAR --runThreadN 4 --runMode alignReads --genomeDir /Applications/stardir/GRCh38genomedir/ --readFilesIn /Applications/stardir/retina_gse40524/SRR548611/SRR548611_1.fastq /Applications/stardir/retina_gse40524/SRR548611/SRR548611_2.fastq  --outFileNamePrefix SRR548611-GRCh38 --sjdbGTFfile /Applications/stardir/gencode.v22.annotation.gtf --outSAMtype BAM SortedByCoordinate --quantMode GeneCounts
Jul 16 10:22:46 ..... Started STAR run
Jul 16 10:22:46 ..... Loading genome
Jul 16 10:24:25 ..... Processing annotations GTF
Jul 16 10:24:44 ..... Inserting junctions into the genome indices
Jul 16 10:27:15 ..... Started mapping
Killed: 9

c2b2afmd2:stardir friedman$ ./STAR --runThreadN 4 --runMode alignReads --genomeDir /Applications/stardir/GRCh38genomedir/ --readFilesIn /Applications/stardir/retina_gse40524/SRR548611/SRR548611_1.fastq /Applications/stardir/retina_gse40524/SRR548611/SRR548611_2.fastq  --outFileNamePrefix SRR548611-GRCh38

Jul 16 10:45:46 ..... Started STAR run
Jul 16 10:45:46 ..... Loading genome
Jul 16 10:47:09 ..... Started mapping
Killed: 9


Thank and best wishes,
Rich

[Alexander Dobin]

Hi Rich,

I think you are working on the brink of RAM requirements. For the bam-sorting case, the alignments were finished, and the problem happened at the sorting stage.

Without sorting, the alignments were not completed.

A few more thing to try:

1. Check your RAM usage, with either top or "Activity Monitor". I seems to me that you have <32GB of RAM available, maybe some processes are eating up your RAM.

2. Try running with one thread. Each tread uses ~150MB of RAM by default. You can reduce this by setting 

--limitIObufferSize 50000000

3. For BAM sorting, limit the RAM with --limitBAMsortRAM 20000000000

Cheers

Alex

[Raviv Pavel]

Hi Alex,

I'm having the same problem as Rich, with similar genome size and resulting log.

I'm trying to run the alignment on a Mac Book Pro, with 16GB of RAM and getting the Killed:9 message when running two threads.

What would be the recommended thread count and limitIObufferSize for my machine? is it even feasible? How long can I expect it to take?

Also, what would be the ideal machine type (CPUs and RAM) if I'm running this on Amazon EC2?

Thanks,

Raviv.

[Richard Friedman]

Alex,

I thought I posted what I am about to post n ow yesterday, but the reply did not go through.
I may have clicked the wrong button,
I had tried what you sugggested without success.
Here are the runs both without and with BAM sorting.

Last login: Thu Jul 16 19:45:15 on console
c2b2afmd2:stardir friedman$ ./STAR --runThreadN 1 —runMode alignReads --genomeDir /Applications/stardir/GRCh38genomedir/ --readFilesIn /Applications/stardir/retina_gse40524/SRR548611/SRR548611_1.fastq /Applications/stardir/retina_gse40524/SRR548611/SRR548611_2.fastq  --outFileNamePrefix SRR548611-GRCh38 --limitIObufferSize 50000000
Jul 20 08:58:21 ..... Started STAR run
Jul 20 08:58:21 ..... Loading genome
Jul 20 09:00:38 ..... Started mapping
^c2b2afmd2:stardir friedman$ ./STAR --runThreadN 1 —runMode alignReads --genomeDir /Applications/stardir/GRCh38genomedir/ --readFilesIn /Applications/stardir/retina_gse40524/SRR548611/SRR548611_1.fastq /Applications/stardir/retina_gse40524/SRR548611/SRR548611_2.fastq  --outFileNamePrefix SRR548611-GRCh38 --limitIObufferSize 50000000 -outSAMtype BAM SortedByCoordinate --limitBAMsortRAM 20000000000

I am attaching both output files and a pdf of actiivty monitor before I started running.
I had shut down all applications of which I was aware but there are some programs
running because of the OS.

QUESTION: There si a 2 Gb SAM file output from the bam run. Is it a complete alignment which
I can turn into a bam file with rsam tools or some other tools?

Also, do you have any further suggestion for running rna-star on a 32Gb Mac?

Thanks and best wishes,
Rich

[Richard Friedman]

Attachments from last message

[Richard Friedman]

atatchment #2

[Richard Friedman]

Alex,

The activity monitor pdf exceeded the group limits so I could not send it.
But I think that you gte the picture

Best wishes,
Rich

[Alexander Dobin]

Hi Rich,

I still cannot understand why the job is killed, could you try to send the activity monitor snapshot by private e-mail. Or convert it to png, which should be a very small file. We need to see how much RAM you have available. Also, please send me the output of `ulimit -a`.

If nothing works, you would need to use sparse suffix array option, --genomeSAsparseD 2 at the genome generation step. This should reduce the RAM requirement to ~16 GB.

Cheers

Alex

[Alexander Dobin]

Hi Raviv,

to run under 16GB, you would need to use --genomeSAsparseD 2 --genomeSAindexNbasesat 12 the genome generation step, and --limitIObufferSize 50000000 at the mapping stage.

Any Amazon machine that has >=32GB of RAM will work, e.g. m4.2xlarge, r3.2xlarge, c3.8xlarge etc.

I would not recommend more than 20 CPUs. as the speed will most likely be limited by the disk speed, which makes the machines with SSDs preferrable.

Cheers

Alex

[saeedeh]

Hi

I am running star on my mac with 16GB RAM and using latest version of human ref file. indexing done successfully  but : 

 ./STAR --genomeDir STAR_Human --readFilesIn STAR_Human/HOS_IL34_1_GATCAG_L005_R1_001.fastq STAR_Human/HOS_IL34_1_GATCAG_L005_R1_001.fastq --outFileNamePrefix test --outSAMstrandField intronMotif 

May 11 11:47:26 ..... started STAR run

May 11 11:47:26 ..... loading genome

May 11 11:57:31 ..... started mapping

Killed: 9

please see the attache log file. 

Regards, Saeedeh

[Alexander Dobin]

Hi Saeedeh,

please send me the Log.out file from the index generation step.

With default parameters, you need ~32GB for the human genome.

For 16GB, you would need to use --genomeSAsparseD 2

Cheers

Alex

[pbpa...@gmail.com]

Hi,

I also have the same problem. I am using star 2.5.3a (bioconda) on MAC 32 GB 1066 MHz DDR3 ECC. Its human data.

Gave me error Killled 9!!

Any solutions??

Thanks,
Payal

On Tuesday, July 7, 2015 at 6:34:15 PM UTC-4, Richard Friedman wrote:

[Alexander Dobin]

Hi Payal,

this likely indicates the insufficient RAM for STAR.

Please send me Log.out file from this run.

Cheers

Alex