Help linking files in the exec command

Samy

unread,

Apr 11, 2019, 1:23:47 PM4/11/19

to singularity

Hello everyone,

I have a container with a binary that relays on input files on the host. I need to link the input files from the local host to the container. when i do:

singularity exec -B $storage/bench_2.5km:/benchmark image.simg ln -s /benchmark/namelist.input /wrf/WRFV3/run/namelist.input ln -s /benchmark/wrfbdy_d01 /wrf/WRFV3/run/wrfbdy_d01 ln -s /benchmark/wrfrst* /wrf/WRFV3/run/wrfrst_d01_2005-06-04_09:00:00 ls /wrf/WRFV3/run

I get the error of:

ln: failed to create symbolic link ?/wrf/WRFV3/run/namelist.input?: File exists

ln: failed to create symbolic link ?/wrf/WRFV3/run/wrfbdy_d01?: File exists

ln: failed to create symbolic link ?/wrf/WRFV3/run/ln?: File exists

But the files are not linked and are not there. I verified with:

singularity exec -B $storage/wrf/bench_2.5km:/benchmark wrf.avx512.simg ls -l /wrf/WRFV3/run

I will be running this on multinode with mpi. Any idea how to link files and run the binaries correctly with the exec command?

I also tried to link these files in the %environment section but that doesn't look like it runs when i use the exec command.

Thank you,

v

unread,

Apr 11, 2019, 2:41:09 PM4/11/19

to singu...@lbl.gov

Why are you having the exec create the links instead of doing direct binds? Since you are creating links on the host, especially with something like MPI you are going to hit the error that the links already exist.

Why not do a bind of each file instead? E.g., here is the start of a command:

singularity exec -B $storage/bench_2.5km/namelist.input:/wrf/WRFV3/run/namelist.input \

-B $storage/bench_2.5km/wrfbdy_d01:/wrf/WRFV3/run/wrfbdy_d01 \

image.simg ls /wrf/WRFV3/run

Why do you have the internal /benchmark directory serving as a middle man when you ultimately are just linking to elsewhere? It doesn't make sense to make symbolic links because you would need write (for inside the container) but then obviously this means needing to bind to the host, and binding to the host (with multiple instances of it going) is going to try to create the same link more than once. TLDR: Try an approach that can handle concurrency - one with binds. I'm not sure if this helps, but it's something to try! Does anyone else have a suggestion?

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

Samy

unread,

Apr 11, 2019, 3:31:46 PM4/11/19

to singularity

Hello Vanessa,

Binding the <local>bench_2.5 to the /benchmark inside the container was the only way that worked . I've tried the way you recommended but for some reason, the container can't see the file from the outside the directory:

$ singularity exec -B $storage/bench_2.5km/namelist.input:/wrf/WRFV3/run/namelist.input wrf.avx512.simg ls /wrf/WRFV3/run

ERROR : Failed to get real path of /usr/local/var/singularity/mnt/final/wrf/WRFV3/run/namelist.input: No such file or directory

ABORT : Retval = 255

On Thursday, April 11, 2019 at 11:41:09 AM UTC-7, vanessa wrote:

Why are you having the exec create the links instead of doing direct binds? Since you are creating links on the host, especially with something like MPI you are going to hit the error that the links already exist.
Why not do a bind of each file instead? E.g., here is the start of a command:

singularity exec -B $storage/bench_2.5km/namelist.input:/wrf/WRFV3/run/namelist.input \
-B $storage/bench_2.5km/wrfbdy_d01:/wrf/WRFV3/run/wrfbdy_d01 \
-B $storage/bench_2.5km/wrfbdy_d01:/wrf/WRFV3/run/wrfbdy_d01 \
image.simg ls /wrf/WRFV3/run

Why do you have the internal /benchmark directory serving as a middle man when you ultimately are just linking to elsewhere? It doesn't make sense to make symbolic links because you would need write (for inside the container) but then obviously this means needing to bind to the host, and binding to the host (with multiple instances of it going) is going to try to create the same link more than once. TLDR: Try an approach that can handle concurrency - one with binds. I'm not sure if this helps, but it's something to try! Does anyone else have a suggestion?

On Thu, Apr 11, 2019 at 1:24 PM Samy <smahan...@gmail.com> wrote:

Hello everyone,

I have a container with a binary that relays on input files on the host. I need to link the input files from the local host to the container. when i do:

singularity exec -B $storage/bench_2.5km:/benchmark image.simg ln -s /benchmark/namelist.input /wrf/WRFV3/run/namelist.input ln -s /benchmark/wrfbdy_d01 /wrf/WRFV3/run/wrfbdy_d01 ln -s /benchmark/wrfrst* /wrf/WRFV3/run/wrfrst_d01_2005-06-04_09:00:00 ls /wrf/WRFV3/run

I get the error of:
ln: failed to create symbolic link ?/wrf/WRFV3/run/namelist.input?: File exists
ln: failed to create symbolic link ?/wrf/WRFV3/run/wrfbdy_d01?: File exists
ln: failed to create symbolic link ?/wrf/WRFV3/run/ln?: File exists

But the files are not linked and are not there. I verified with:
singularity exec -B $storage/wrf/bench_2.5km:/benchmark wrf.avx512.simg ls -l /wrf/WRFV3/run

I will be running this on multinode with mpi. Any idea how to link files and run the binaries correctly with the exec command?

I also tried to link these files in the %environment section but that doesn't look like it runs when i use the exec command.

Thank you,

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

v

unread,

Apr 11, 2019, 4:54:24 PM4/11/19

to singu...@lbl.gov

Are you able to package up small set of flies and provide your container recipe so others can reproduce it, and help? I'll offer to help if you are able to do this.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

Reply all

Reply to author

Forward