Running out of space w/AWS Batch

[stephen mclaughlin]

Hi there,

    I understand that Managed Compute Environments launch the ECS Optimized AMI which includes an 8GB volume for the operating system and a 22GB volume for Docker image and metadata storage.  We made our own image using that ECS Optimized AMI and upgraded the amount of storage for the Docker image and metadata storage to 1TB, but I'm still getting error messages that I'm running out of disk space trying to run a container that performs trim_galore on a pair of ~7.5Gb compressed FASTQ files.  It seems that nextflow grabs the files it needs off of S3 from within the container via an aws s3 cp command; is that right?  I'm also logging into the instance while this is running and monitoring the disk usage using docker info | grep Space and it seems to die without getting anywhere close to the 1TB upper limit; I'm not even sure it is managing to finish copying the files over before it dies.  The Docker image itself is about 3GB and this will die without any other jobs running on that instance.  I am including .nextflow log.  

Thanks,

Stephen

[Paolo Di Tommaso]

The docker storage configuration is quite tricky. Make sure to have followed the explanation in the documentation, in particular you will need to set the docker volume base size (see the *warning* yellow box).

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[stephen mclaughlin]

thanks!  this answer solved my problem.

Best,

Stephen

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.