Nexflow inside a native app and access to private docker repository

[Andrea Argentini]

Hi Illumina team,

I am developing a native app in BaseSpace , based on a nextflow workflow. Here are my question : 

- Nextflow needs a sort of staging area (work directory) used to process the data in each step of the pipeline. The space required by this  staging area could increase a lot (this also depends on the number of samples in input). I was wondering if I could use the disk space associated with ec2 instances or there are other possibilities (e.g a sort of aws bucket in BaseSpace)  If it is possible to use only the disk space  how is the biggest space available ?  here I have found that EC2 instance has 4 x 840 GB ?

 

-  I have created a private docker repository, following the steps listed in the documentation  where to store the docker image. Everything worked fine, I created the repository and I was able to pull and pull the image (using sudo docker pull docker.illumina/repository_name). Then in the callback.js I specified containerImageId: "docker.illumina.com/repository_name" but then when I launch the project I get the error 'Error downloading application image id docker.illumina.com/repository_name'. Since the docker repository is private, I think I have to to prompt with my credential (or store them somewhere) as I do with sudo pull but in the documentation but I haven't found any mention of that or neither an example. Can you help me with that or point me out to right direction to overcome this problem ?

[Stefano Berri]

Hi.

- You can store large temporary files in `/data/scratch/` as mentioned in the documentation . I am not sure how large that is, but it is quite large and - I think - it is "local" to the VM, so quite fast IO
- In the callback you should specify the image like this: `containerImageId: "docker.illumina.com/NAMESPACE/NAME@digest"` where digest is the sha256 digest.

I hope this will help you