"No such file or directory" error

[Zack]

Hello, 

# I am running a docker image inside of a singularity container on my university's cluster. The program seems to be running just fine, but only makes it partway through then fails with this message:

No such file or directory at /opt/scripts/common/fragment.pl

# However, I can check that the script exists in the appropriate location using this command:

singularity exec juicer_new.img ls /opt/scripts/common

# This is how I am running the software in singularity using a SLURM script:

#!/bin/bash

#SBATCH --job-name=juicer_singularity

#SBATCH -p general

#SBATCH -N 1

#SBATCH -c 20

#SBATCH --mem-per-cpu=6G

#SBATCH -t 02:00:00

module load Singularity/2.4.2

singularity run juicer_new.img \

    -d /ysm-gpfs/home/zrl3/scratch60/singularity \

    -z /ysm-gpfs/home/zrl3/scratch60/singularity/references/Homo_sapiens_assembly19.fasta \

    -y /ysm-gpfs/home/zrl3/scratch60/singularity/restriction_sites/hg38MboI.txt \

    -p hg38 -s MboI

# I ran the script and included the --verbose flag. The output is attached. Could someone offer any suggestions for troubleshooting why this "No such file or directory" error is occurring? Getting the software to run requires some large datasets, making it difficult to produce a reproducible example. For reference, I created the container using this command:

singularity pull --name juicer_new.img docker://index.docker.io/aidenlab/juicer

[v]

Hey Zack,

I can offer to help to troubleshoot, even if I don't know the answer right off the bat, maybe some questions will help to start! it looks like the entrypoint to your script is this:

#!/bin/sh
exec "juicer.sh" "-D" "/opt"

and that you are also passing parameters from the slurm submission (-z, -d, -y and -s and -p) (haha, S&P) but then in the script itself that does the exec (above) you don't have any way to catch them. Have you tried:

#!/bin/sh
exec "juicer.sh" "-D" "/opt" "$@" 

As a first effort? Maybe some of your preferences aren't going in correctly? Could you please run again with --debug so we can also see if any mounts are being done so that /opt isn't what we think it is?

Best,

Vanessa 

 

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Zack]

Hi Vanessa,

Many thanks for your suggestions! I have attached the output from --debug.

I am not quite sure that I follow your first suggestion, though. During job submission I submit the SBATCH script using sbatch <name of job submission script>. I understand what you are saying about how the script won't see the parameters passed to SLURM and I understand that "$@" should pass along the user arguments, but I'm not sure where this option should be included. I did try to shell into the image and tried to execute juicer.sh, which appears to work (as shown below), but it is not finding the required sequence data located in a parent directory. If this is not what you were suggesting, please let me know. 

singularity shell -c juicer_new.img

Singularity: Invoking an interactive shell within container...

Singularity juicer_new.img:~>  #!/bin/sh

Singularity juicer_new.img:~> exec "juicer.sh" "-D" "/opt" "$@" 

***! Reference sequence /opt/references/Homo_sapiens_assembly19.fasta does not exist

Let me know if you have other suggestions. Thanks again,

Zack

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Zack]

Just an update: I haven't been able to solve this problem. I have tried re-building the image in several ways and am still hitting the same missing directory error. Any suggestions would be very much appreciated. Thanks in advance!

[david hon]

Hi Zach,

Question -- does the Singularity runtime use a bind-mount of /opt or ...

I understand you "imported" a Docker image to create the Singularity image;

did you also "import" all /opt/* data into the Singularity image?

Presumably when using the Docker runtime on the original Docker image, /opt

was bind-mounted via "docker run -v /opt:/opt ..."

Sorry but I'm new to Singularity, and this is helping me understand it better.

I understand Singularity automatically bind-mounts one's /home/user-account

and the current-working-directory ... can it bind-mount /opt to (via CLI like Docker)?

Cheers,

--david

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

apologies if my text lacks upper-case -- the shift
key aggravates my carpal tunnel & tendinitis.
david...@gmail.com cell: 352-275-7438

[David Godlove]

Hi Zach,

Singularity has a --bind/-b argument that can be used with all action subcommands.  It's also possible to set the $SINGULARITY_BINDPATH variable to bind host directories into the container. 

Dave

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

apologies if my text lacks upper-case -- the shift
key aggravates my carpal tunnel & tendinitis.
david...@gmail.com cell: 352-275-7438

[david hon]

Hi Dave, Zach,

Cool -- so assuming Zach has no typo in his --bind or -b Singularity CLI usage, one would think the only remaining issue is /opt host file-system permissions ...

(assuming Zach is running the Singularity container in a non-sudoer user account).

Cheers,

--david

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.