Trying to understand the shared memory usage for AWS

[Ryan Golhar]

Hi all - I'm trying to understand how the shared memory usage for STAR works, especially in an AWS ec2 environment. 

I was initially trying to use a c3.2xlarge instance type and aligning to hg19, but c3.2xlarge only provides 15GB of RAM.  I'm since move to a c3.4xlarge with provides 30GB of RAM. 

I'm using 8 threads per STAR alignment (16 cores available on this machine) so I figure I can run two STAR jobs at the same time.  When I use the --genomeLoad LoadAndKeep option and both STAR alignments start at approximately the same time, how will the genome index be loaded into memory?  Will one wait or will both try to load?  I didn't find enough information about this in the docs.  Should I load the genome first on all my ec2 instances before aligning anything?  What's the recommended approach for this?

Ryan

[Ryan Golhar]

Let me clarify....I have a node with 30GB of RAM (c3.4xlarge) and I do:

1.  STAR --genomeLoad Load
2.  STAR --genomeLoad LoadAndKeep --readFilesIn...

The alignment job starts properly.  I then do another alignment job.  When I do this also with LoadAndKeep.  As soon as the second job starts, the first one is killed, but the second continues.  What am I missing?  This is STAR version 2.4.1d

[Alexander Dobin]

Hi Ryan,

if one job is loading genome into the shared memory, the other will be waiting for the loading to complete.

You can check it by looking at the messages at the end of the Log.out file.

If two jobs are submitted exactly at the same time, it could, theoretically lead to a racing condition, with both jobs loading the genomes and overflowing RAM.

I would recommend a short pause between the jobs, say 1 second.

Another possibility is that the run-time I/O buffers are overflowing the RAM when two jobs are running. You can try to reduce the usage by reducing --limitIObufferSize from the default 150000000 (i.e. ~150 MB) to 50000000 (i.e. 50MB). Note that this is per thread value, so the total size of the buffer can be quite large.

Cheers

Alex

[justin joseph jeyakani]

Hi Alex,

Could you please explain me how the genomeLoad in shared memory works in the below case?

The jobs are used in core facility production environment, when the sequencing completed with STAR mapping required..

Version used:STAR_2.4.1d

Let's say the script called by project-A which need to submit 50 jobs to node1 on certain queue (rnaseq.q) which allow four jobs (4*20 slots) to run at a time.

1.  STAR --genomeLoad LoadAndExit

2.  sleep 1

2.  STAR --genomeLoad LoadAndKeep --readFilesIn sample-A.fastq

    STAR --genomeLoad LoadAndKeep --readFilesIn sample-B.fastq

    STAR --genomeLoad LoadAndKeep --readFilesIn sample-C.fastq

    STAR --genomeLoad LoadAndKeep --readFilesIn sample-D.fastq

when the above jobs finished the subsequent jobs will enter the queue and at the end

3.STAR --genomeLoad Remove

As you mentioned in this thread "sleep 1" will help to prevent any of the step two jobs (SampleA,SampleB,sampleC,sampleD) to load the genome at the same time when step 1 happening.

For a case when the 49th and 50th job running in the node1 (48 samples job completed and node have two free slot to accept two more jobs to run), another set of job submitted by the script for project-B which contain 20 jobs.

In the available 2 slots one slot would be taken up by "STAR --genomeLoad LoadAndExit" and the another slot will be taken up by Sample-AA of project-B for mapping which uses the genome already loaded by the project-A and before 

this sample mapping complete, if the project-A's 49th and 50 sample mapping completed and procedd to "STAR --genomeLoad Remove" would affect the undergoing the mapping of sample-a of project-B right?

Also can I know the difference btn "LoadAndKeep" and "LoadAndExit" or both can be used to only loading the genome? and

--limitIObufferSize from the default 150000000 (i.e. ~150 MB) to 50000000 you mentioned for situation like loading the genome at the same time only or it may be needed for during the mapping also if any parallel jobs?

[Alexander Dobin]

Hi Justin,

STAR --genomeLoad LoadAndExit does run only to load the genome, and then exits, so it will not be taking any slots.

If project-B starts the jobs before the STAR --genomeLoad Remove command, then it will run on the available slots and use the same genome loaded in the shared memory.

When STAR --genomeLoad Remove runs, it marks the shared memory genome for destruction, but does not remove it until all the jobs attached to it exit.

You do not really need to use STAR --genomeLoad LoadAndExit. You can do directly 

STAR --genomeLoad LoadAndKeep --readFilesIn sample-A.fastq

sleep 1

STAR --genomeLoad LoadAndKeep --readFilesIn sample-B.fastq

STAR --genomeLoad LoadAndKeep --readFilesIn sample-C.fastq

The --genomeLoad LoadAndKeep option loads the genome if it's not already loaded, and then starts mapping.

--limitIObufferSize is the size of the RAM reserved for IO buffers, per thread! So, if you are running multiple STAR jobs which use 80 threads on your node, it will take extra 150MB*80=12GB of RAM, on top of the genome size  (since the genome is shared among all threads, it will only require RAM equal to one genome size).

Cheers

Alex

On Thursday, January 21, 2016 at 10:53:56 AM UTC-5, justin joseph jeyakani wrote:

Hi Alex,

Could you please explain me how the genomeLoad in shared memory works in the below example?

The below jobs used in core facility production environment, whenever the sequencing is completed with STAR mapping required..

Version:STAR_2.4.1d

Let's say the script called by project-A which need to submit 50 jobs to node1 on certain queue (rnaseq.q) which allow four jobs (4*20 slots) to run at a time.

1.  STAR --genomeLoad LoadAndExit

2.  sleep 1

3.  STAR --genomeLoad LoadAndKeep --readFilesIn sample-A.fastq

    STAR --genomeLoad LoadAndKeep --readFilesIn sample-B.fastq

    STAR --genomeLoad LoadAndKeep --readFilesIn sample-C.fastq

    STAR --genomeLoad LoadAndKeep --readFilesIn sample-D.fastq

whenever the above jobs finished the subsequent jobs will enter the queue ...at the end

4.STAR --genomeLoad Remove

As you mentined in this thread "sleep 1" will help to prevent any of the step two jobs to (SampleA,SampleB,sampleC,sampleD) load the genome at the same time when step 1 happening.

For a case when the 49th and 50th job running in the node1 (48 samples job completed and node have two free slot to accept two more jobs to run), another set of job submitted by the script for project-B which contain 20 jobs.

In the available 2 slots one slot would be taken up by "STAR --genomeLoad LoadAndExit" and the another slot will be taken up by Sample-AA of project-B for mapping which uses the genome already loaded by the project-A and before 

this sample mapping complete if the project-A's 49th and 50 sample mapping completed and procedd to "STAR --genomeLoad Remove" would affect the undergoing the mapping of sample-a of project-B right?

Also can I know the difference btn "LoadAndKeep" and "LoadAndExit" or both can be used to for only loading the genome?

And --limitIObufferSize from the default 150000000 (i.e. ~150 MB) to 50000000 you mentioned for situation like loading the genome at the same time only or it may be needed for during the mapping if parallel jobs?

[justin joseph jeyakani]

Hi Alex,

Thank you for your reply.

This is wonderful as long the the loaded genome in the shared memory is not removed when any jobs are using it. Do you think this is violation in cluster, when my jobs access the other users already loaded genome in a memory if any? 

I have another query about sorting the bam? when I use the shared memory option how much RAM would be desired "--limitBAMsortRAM"

My jobs are failed ...sorting bam state not enough memory (Max memory needed for sorting = 16516456) but the cluster have it at that time, I used outBAMsortingThreadN 20 and limitBAMsortRAM 20 the genome is human.

Cheers,

Justin

[justin joseph jeyakani]

Hi Alex,

the --limitBAMsortRAM need to be given as bytes? if I give 20 it fails but work for 2016346648.

One more query here..

Could you please explain me the mechanism of how the shared memory works in STAR? does this have any Id to identify or specify? and

Other than parallel/sequential jobs access the  loaded genome in the shared memory by one user could be accessed (read) by another user on the same node or restricted to user permission?

Cheers,

Justin

On Monday, January 25, 2016 at 4:13:12 AM UTC+8, justin joseph jeyakani wrote:

Hi Alex,

Thank you for your reply.

This is wonderful as long the the loaded genome in the shared memory is not removed when any jobs are using it. Do you think this is violation in cluster, when my jobs access other users already loaded memory if any? 

I have another query about sorting the bam..when I use the shared memory option how much RAM would be desired "--limitBAMsortRAM"

My jobs are failed when sorting bam stating not enough memory (Max memory needed for sorting = 16516456) but the cluster have it at that time, I used outBAMsortingThreadN 20 and limitBAMsortRAM 20 the genome is human.

[Alexander Dobin]

Hi Justin,

--limitBAMsortRAM is in bytes, so 2016346648 is ~2GB

Note that with genome in shared memory, this allocation is in addition to the genome, and it is not shared between jobs on the same node, so it's easy to overflow RAM if you have many jobs on the same node.

The genomes in shared memory are uniquely identified by their directories (absolute path). The permission for shared memory is set to 666, i.e. all users have full access to any shred genome.

Cheers

Alex

[justin joseph jeyakani]

Hi Alex,

Thank you for addressing all my queries.

Cheers,

Justin