simple exec failed

[mahedi]

Hello,

i'm trying this tool and, as a newbie, it fail..

i compile it from source ..

Where is my fault?
$ singularity specgen /bin/echo

WROTE: echo.sspec

$ singularity build ./echo.sspec
Creating temporary container path...
Singularity App Container Name: echo.sapp
Running build scriptlet
Evaluating: packages
Building the runtime level: 1
Installing file: /bin/sh
Installing file: /lib/x86_64-linux-gnu/libc.so.6
Installing file: /lib64/ld-linux-x86-64.so.2
Installing file: /lib64/libnss_files.so.2
Installing file: /lib32/libc.so.6
Installing file: /lib/ld-linux.so.2
Evaluating: Exec
Evaluating: files
Installing file: /bin/echo
Installing file: /usr/lib/locale/locale-archive
Running install scriptlet
Creating template configs
  /etc/nsswitch.conf
  /etc/resolv.conf
  /etc/hosts
Checking for files that need santizing
Found locale-archive, removing...
Cleaning up device files
Running post scriptlet
Defining container shell envrionment
Building singularity main runscript
Building singularity test runscript
Building singularity exec runscript
Running test...
/singularity: 1: /.singularity.env: declare: not found
/singularity: 2: /.singularity.env: declare: not found
/singularity: 3: /.singularity.env: declare: not found
/singularity: 4: /.singularity.env: declare: not found
Including specfile
Writing metadata
Creating output SAPP container
WROTE: echo.sapp
$ ls
echo.sapp  echo.sspec

$singularity install echo.sapp
Installing: echo (f6221fa8-aa69-458c-8e2c-5168f5107b13)
/usr/local/libexec/singularity/cli/install.exec: 61: exec: 99: not found

$./echo.sapp
/usr/local/libexec/singularity/cli/run.exec: 61: exec: 99: not found

and in debug mode :
$ singularity -d run ./echo.sapp
...
 OUT=UUID: f6221fa8-aa69-458c-8e2c-5168f5107b13
+ echo UUID: f6221fa8-aa69-458c-8e2c-5168f5107b13
+ head -n 1
+ sed -e s@^uuid:\s*@@i
+ return 0
+ SAPPUUID=f6221fa8-aa69-458c-8e2c-5168f5107b13
+ LOCKFILE=/home/mahedi/.singularity-cache/locks/f6221fa8-aa69-458c-8e2c-5168f5107b13.lock
+ exec 99
/usr/local/libexec/singularity/cli/run.exec: 61: exec: 99: not found


Can you help me...

Thanks

[Gregory M. Kurtzer]

Oh, that is very interesting! What distribution of Linux are you running?

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Gregory M. Kurtzer
Technical Lead and HPC Systems Architect
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

[mahedi]

it's lubuntu 15.10

[Gregory M. Kurtzer]

Ohh, I've never used that before. What does it use for /bin/sh... I gather it is not bash because bash (and several other shells) have exec as a built in function.

What is the output of these commands:

$ ls -l /bin/sh

$ which exec

Until I fix this, I think you can take the output of the above command and add that to a files section of the config... For example:

%files

/bin/exec

Then rebuild your SPEC, and hopefully it will just work.

[mahedi]

ok i see :
ls -l /bin/sh
lrwxrwxrwx 1 root root 4 juin   4  2015 /bin/sh -> dash
$ which exec
$
:-)


no exec at all ...

[Gregory M. Kurtzer]

Hrmm, I am pretty certain that dash has an internal function for exec. Hrmm, but it also seems that "declare" might be my mistake in confusing bash syntax for Bourne.

What happens if you run the command:

$ sh -c "exec sleep 1"

Can I see the output of the following commands:

$ singularity delete echo

$ singularity -d install echo.sapp

Sorry it didn't work right off the bat for you, I've never tested nor ran Lubuntu before thus this feedback and debugging is very valuable!

Thanks! 

[mahedi]

no problem at all. I 'm happy to help you

$ sh -c "exec sleep 1"

$ singularity delete echo
$ singularity -d install echo.sapp

enabling debugging
ending argument loop
+ load_mod cache
+ MOD=cache
+ [ -z /usr/local/libexec ]
+ [ -f /usr/local/libexec/singularity/mods/cache.smod ]
+ . /usr/local/libexec/singularity/mods/cache.smod
+ load_mod sappfile
+ MOD=sappfile
+ [ -z /usr/local/libexec ]
+ [ -f /usr/local/libexec/singularity/mods/sappfile.smod ]
+ . /usr/local/libexec/singularity/mods/sappfile.smod
+ HEADER_END=SINGULARITY_END_HEADER
+ export HEADER_END
+ UUID_DIR=containers
+ NAME_DIR=names
+ LOCK_DIR=locks
........
+ OUT=UUID: f6221fa8-aa69-458c-8e2c-5168f5107b13

+ echo UUID: f6221fa8-aa69-458c-8e2c-5168f5107b13
+ head -n 1
+ sed -e s@^uuid:\s*@@i
+ return 0
+ SAPPUUID=f6221fa8-aa69-458c-8e2c-5168f5107b13
+ LOCKFILE=/home/mahedi/.singularity-cache/locks/f6221fa8-aa69-458c-8e2c-5168f5107b13.lock
+ exec 99

/usr/local/libexec/singularity/cli/install.exec: 61: exec: 99: not found
$

same as before..

is here a difference between TAR file and git master branch where i got the source?

best regards

[Gregory M. Kurtzer]

Ohh, very interesting. The problem isn't exec, the problem is that exec is being called with an integer...

The tar file referenced on the github.io pages pulls from the 1.0 branch instead of master. You might want to test with the 1.0 branch, but I'm not sure it will make much of a difference. I will take a look at this and try to debug shortly and let you know.

Thanks!

[mahedi]

you're right .. same issue with tar file..

Thanks

[Gregory M. Kurtzer]

Can you do a git pull from master, and try again? I just committed something that I'm hoping will help.

[Gregory M. Kurtzer]

Do another git pull if you already did a pull... Jason just let me know that Dash also doesn't like a multi-digit file descriptor number, so I changed it to just 9 and just committed.

Thanks again!

[mahedi]

OK ... now it's different..

after install and building specgen echo

$ singularity run ./echo.sapp

/singularity: 1: /.singularity.env: declare: not found
/singularity: 2: /.singularity.env: declare: not found
/singularity: 3: /.singularity.env: declare: not found
/singularity: 4: /.singularity.env: declare: not found

$ ./echo.sapp

/singularity: 1: /.singularity.env: declare: not found
/singularity: 2: /.singularity.env: declare: not found
/singularity: 3: /.singularity.env: declare: not found
/singularity: 4: /.singularity.env: declare: not found

$

[Gregory M. Kurtzer]

OK, good news on the first issue... Hopefully I got the second one now... Will do you another git pull and rebuild everything?

[marc sert]

ok it's working now !

thanks a lot...

but... there is an unwanted whitespace at the beginning of echo...
$ singularity run echo.sapp "aaaaabbbbbb"
aaaaabbbbbb
$ /bin/echo "aaaaabbbbbb"
aaaaabbbbbb
$

any idea?

[marc sert]

and excuse me for the delay.... it's early here (france, near bordeaux...)

best regards

--
marc...@gmail.com
GnuPG: 52B8CD41

[Gregory M. Kurtzer]

Hi Marc!

That is good progress, excellent!

Does your specfile just have "Exec: /bin/echo" in it?

If so can you send the output of:

$ singularity strace echo

Note: you don't need to reference the .sapp file once it has been cached as can be seen by "singularity list".


Sent from my iPhone

[mahedi]

Hi gregory!

ok it's generated by specgen

Name: echo
Summary: Generated by 'singularity specgen'
Maintainer:
DebugOS: 1

%runscript
exec /bin/echo "" "$@"


%test


%files
/bin/echo
/lib/x86_64-linux-gnu/libc.so.6
/usr/lib/locale/locale-archive
----------------------
$ singularity strace echo
Strace is not found within the container...
$ strace echo
execve("/bin/echo", ["echo"], [/* 40 vars */]) = 0
brk(0)                                  = 0xcc4000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
....

[Gregory M. Kurtzer]

Ahhh, the extra space I think is from the %runscript... Notice the first quote.

The specgen function just provides a template specfile so feel free to edit it all you want!

BTW, the locale-archive is usually pretty big and can be removed if you don't mind losing your locales.

Now... I'm curious why strace wasn't found and included in your container. Can you send me the output of:

$ which strace

Thanks again, your feedback and help on this is very valuable!

[mahedi]

Hello,
$ which strace
/usr/bin/strace
$ strace -V
strace -- version 4.8

[Gregory M. Kurtzer]

At this point, everything (aside from the strace) seems to be working, correct?

To debug strace, can you send me the build output:

$ singularity build echo.sspec

Can you also do one more thing for me. I'd like to see all of the files in the 'echo.sapp' container. Can you first do a "singularity list" and look for the UUID for the container named 'echo' (if there are multiple echo's installed, use the lowest one). Then take that UUID and do the following:

$ find ~/.singularity-cache/containers/[UUID]/c -type f

Thanks again!

[marc sert]

2016-03-02 22:50 GMT+01:00 Gregory M. Kurtzer <gmku...@lbl.gov>:
> singularity build echo.sspec

ok files following

and look at bc error in :

du -sk ...../.singularity-cache/containers/ca5d2c7e-d80b-4a6d-8a46-be2230b67305/c
+ cut -f 1
+ SIZE_K=4292
echo -ne scale=1\n4292/ 1024\n
+ bc
(standard_in) 1: syntax error
+ SIZE_M=4
+ echo 4

with bc -v
bc 1.06.95

$ singularity list
CONTAINER NAME UUID SIZE(Mb) SUMMARY
(standard_in) 1: syntax error
echo ca5d2c7e-d80b-4a6d-8a46-be2230b67305 4
Generated by 'singularity specgen'

best regards




--
marc...@gmail.com
GnuPG: 52B8CD41

[Gregory M. Kurtzer]

Hi Marc,

Ok, I'm crossing my fingers and hoping we got everything fixed now.

Thank you again!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Jason Stover]

Hi Marc,

On an Arch install I was able to build the current (as of an hour or
so ago) git tree and do the following with no errors:

$ singularity specgen /bin/echo
$ singularity install echo.sspec
$ singularity list
$ singularity run echo "Hello World"
$ ./echo.sapp "Hello World"
$ singularity delete echo

Hopefully on your lubuntu install it won't be to much different from
the Arch install that was laying around at the office that I was
testing on.

Thanks,
-J

[Gregory M. Kurtzer]

Hi Marc,

I even went as far as to download Lubuntu and install it in a VM and made sure the test.sh runs properly. Hopefully when you grab the latest commits everything will just work.

Thank you again for your patience and help in troubleshooting!

Jason, thank you as well for running the same tests and debugging on ArchLinux!


Sent from my iPhone

[mahedi]

yes!

everything I report is correct now!

thank you !

[mahedi]

ok thank you too Jason...

next target is .... Alpinelinux with grsec !

have you any idea about correct setup of roles with gradm?

Best Regards

[Gregory M. Kurtzer]

Excellent!!

Sent from my iPhone

[Gregory M. Kurtzer]

Ohhhhh,!!!! I've been curious about Alpine Linux since I heard about it last week at Stanford in a talk about Docker and HPC. 

I will also install that into a VM and test it out, but I think you will beat me as I won't be able to do that until possibly next week. So please keep me informed!!

They seem to use their own packaging tools so maybe we can get Singularity included in the distribution as soon as we get 1.0 released as stable. 

I am curious, what do you do with all these distributions? 

Thanks again!

Sent from my iPhone

--

[Jason Stover]

I don't know if anyone has touched grsec ... I haven't in years, and
that was only playing with it.

From a quick glance, it probably need similar to an lxc/docker role if
you have one, and suexec/suxec/blah! (gmk, why'd you have to name that
one binary what you did! ... can never remember the name) will need to
have CAP_SETUID on it. Beyond that it's probably going to be trial and
error .. and then also dependent on what other roles you have set up
(i.e. socket binding, limits, etc..).

-J

[Gregory M. Kurtzer]

We used to include grsec/PAX enabled kernels with Caos Linux, but we didn't have it enabled aggressively by default. But I think you are right, it may take some playing with in order to get the right balance of security and permission to run properly.

Also I'm curious if we are going to see additional issues with Ash being the default shell as it is based on Busybox.

Hopefully it comes with a complete development toolchain as well.


Sent from my iPhone