Singularity 2.2.99 and warnings with passwd, group

127 views
Skip to first unread message

tin h

unread,
Apr 2, 2017, 3:16:07 AM4/2/17
to singu...@lbl.gov
Hi folks, 

Hope it is not something amiss with my config.  But while testing a build with Singularity 2.2.99, I am getting a number of warnings about missing user and group.  see below for details.


Host: RHEL 6.6
Container: Centos 7

When I ran that Singularity definition file on a system with the dev code for the upcoming Singularity 2.3, I saw some warnings about /etc/group and /etc/passwd.
I reran the same build using 2.2.1 and don't see them.

In the resulting container build by "2.2.99", these number of system users in the passwd file were surprisingly slim:

Singularity topspin.4.img> cat /etc/passwd
systemd-bus-proxy:x:999:997:systemd Bus Proxy:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:998:996:User for polkitd:/:/sbin/nologin
tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin


Anyone else noticed such problem with singularity 2.2.99?

Regards,
Tin



 > sudo -E singularity bootstrap topspin.img Singularity | tee sing.log 2>&1
Building from bootstrap definition recipe
Adding label: 'SINGULARITY_CONTAINER_UUID' = 'ecafe2ee-cc49-43c2-8869-9367d738a2e8'
Adding label: 'SINGULARITY_DEFFILE' = 'Singularity'
Adding label: 'SINGULARITY_BOOTSTRAP_VERSION' = '2.2.99'
Adding label: 'SINGULARITY_DEFFILE_BOOTSTRAP' = 'yum'
Adding label: 'SINGULARITY_DEFFILE_OSVERSION' = '7'
usage: add.py [-h] [--key KEY] [--value VALUE] [--file FILE] [-f]
add.py: error: unrecognized arguments: https
/e/libexec/singularity/bootstrap-scripts/pre.sh: line 64: basearch: unbound variable
Adding base Singularity environment to container
Found YUM at: /usr/bin/yum
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package centos-release.x86_64 0:7-3.1611.el7.centos will be installed
---> Package coreutils.x86_64 0:8.22-18.el7 will be installed
--> Processing Dependency: rtld(GNU_HASH) for package: coreutils-8.22-18.el7.x86_64
--> Processing Dependency: ncurses for package: coreutils-8.22-18.el7.x86_64
--> Processing Dependency: librt.so.1(GLIBC_2.3.3)(64bit) for package: coreutils-8.22-18.el7.x86_64
--> Processing Dependency: libpthread.so.0(GLIBC_2.3.2)(64bit) for package: coreutils-8.22-18.el7.x86_64
--> Processing Dependency: libpthread.so.0(GLIBC_2.2.5)(64bit) for package: coreutils-8.22-18.el7.x86_64




Transaction Summary
================================================================================
Install      91 Package(s)

Total download size: 49 M
Installed size: 246 M
Downloading Packages:
--------------------------------------------------------------------------------
Total                                           5.3 MB/s |  49 MB     00:09
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
^M  Installing : libgcc-4.8.5-11.el7.x86_64                                  1/91
^M  Installing : ncurses-base-5.9-13.20130511.el7.noarch                     2/91
^M  Installing : centos-release-7-3.1611.el7.centos.x86_64                   3/91
^M  Installing : setup-2.8.71-7.el7.noarch                                   4/91
warning: /etc/group created as /etc/group.rpmnew
warning: /etc/hosts created as /etc/hosts.rpmnew
warning: /etc/passwd created as /etc/passwd.rpmnew
     # don't see these files inside the container when done #
^M  Installing : tzdata-2016g-2.el7.noarch                                   5/91
^M  Installing : filesystem-3.2-21.el7.x86_64                                6/91







  Installing : util-linux-2.23.2-33.el7.x86_64                              8/8
warning: group tty does not exist - using root
chown: invalid user: ‘root:root’
  Verifying  : libsemanage-2.5-5.1.el7_3.x86_64                             1/8

...

  Installing : sysvinit-tools-2.88-14.dsf.el7.x86_64                     75/318
warning: group tty does not exist - using root
  Installing : hostname-3.13-3.el7.x86_64                                76/318


...
  Installing : 1:cups-libs-1.6.3-26.el7.i686                            248/318
useradd: group '100' does not exist
useradd: the GROUP= configuration in /etc/default/useradd will be ignored
  Installing : polkit-0.112-11.el7_3.x86_64                             249/318

...

  Installing : initscripts-9.49.37-1.el7.x86_64                         266/318
chown: invalid user: ‘root:utmp’
  Installing : kernel-3.10.0-514.10.2.el7.x86_64                        267/318
  Installing : systemd-sysv-219-30.el7_3.7.x86_64                       268/318
  Installing : ghostscript-cups-9.07-20.el7_3.3.x86_64                  269/318
  Installing : 1:cups-1.6.3-26.el7.x86_64                               270/318
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
warning: user lp does not exist - using root
warning: group sys does not exist - using root
warning: group lp does not exist - using root
warning: user lp does not exist - using root
warning: group sys does not exist - using root
warning: group lp does not exist - using root
warning: group lp does not exist - using root
  Installing : cups-pdf-2.6.1-7.el7.x86_64                              271/318
pgrep: invalid user name: root
  Installing : mdadm-3.4-14.el7_3.1.x86_64                              272/318




On Tue, Mar 21, 2017 at 12:22 PM, tin h <tin...@gmail.com> wrote:

I am happy to report that argparse is no longer needed!
The dinosaur thank you! :-)



On Mon, Mar 20, 2017 at 9:28 AM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:
Hi Tin!

Vanessa sent me a PR to fix this, and I just merged it in. Can you retest the "development" branch on your RH 6 system without the pip installed version of argparse?

Thanks!

On Mon, Mar 20, 2017 at 8:06 AM, tin h <tin...@gmail.com> wrote:
Thanks Vanessa.
I like argparse myself, but less dependency likely lower the bar for adoption.  I think there are more than one dinosaur company out there :-D

Cheers,
Tin

On Sun, Mar 19, 2017 at 11:41 PM, vanessa s <vso...@gmail.com> wrote:
This is my bad - I should have used optparse for it's legacy support in favor of argparse, because this is what we did before:


because argparse was provided in standard library after python 2..7:


I'll make a PR to fix this up - sorry about that tin!


On Sun, Mar 19, 2017 at 11:26 PM, tin h <tin...@gmail.com> wrote:

Nicely done Greg!

FYI,
I got a little issue running the dev branch in RHEL 6.6
singularity bootstrap complains about argparse module not found (see below).

I know I am running old OS that contains old python (python-2.6.6-52) but legacy support is one of the salient feature of Singularity right? :)

Things worked after "sudo -E pip install argparse".  I don't think this was needed in the previous release... perhaps it should be mentioned in the release notes... assuming sys admin actually RTFM :-p


Cheers,
Tin



Traceback (most recent call last):
  File "/e/libexec/singularity/python/helpers/json/add.py", line 37, in <module>
    import argparse
ImportError: No module named argparse
Traceback (most recent call last):
  File "/e/libexec/singularity/python/helpers/json/add.py", line 37, in <module>
    import argparse
ImportError: No module named argparse
Traceback (most recent call last):
  File "/e/libexec/singularity/python/helpers/json/add.py", line 37, in <module>
    import argparse
ImportError: No module named argparse
/e/libexec/singularity/bootstrap-scripts/pre.sh: line 64: basearch: unbound variable
Adding base Singularity environment to container
Found YUM at: /usr/bin/yum
Setting up Install Process
base                                                     | 3.7 kB     00:00
base/primary_db                                          | 4.7 MB     00:00




On Sat, Mar 18, 2017 at 2:25 PM, vanessa s <vso...@gmail.com> wrote:

​

On Sat, Mar 18, 2017 at 4:37 PM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:
Hey everyone,

I just merged the 'lib-refactor' branch into the 'development' branch. From this point on, all pull requests should be targeting the 'development' branch.

Users are encouraged (and asked) to start playing with the 'development' branch and let us know about bugs, issues, and give us feedback. This is important to ensure that we have a stable upgrade path from 2.2 to 2.3.

Thank you!

--
Gregory M. Kurtzer
HPC Systems Architect and Technology Developer
Lawrence Berkeley National Laboratory HPCS
University of California Berkeley Research IT
Singularity Linux Containers (http://singularity.lbl.gov/)
Warewulf Cluster Management (http://warewulf.lbl.gov/)

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.



--
Vanessa Villamia Sochat
Stanford University '16

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.



--
Vanessa Villamia Sochat
Stanford University '16

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.



--
Gregory M. Kurtzer
HPC Systems Architect and Technology Developer
Lawrence Berkeley National Laboratory HPCS
University of California Berkeley Research IT
Singularity Linux Containers (http://singularity.lbl.gov/)
Warewulf Cluster Management (http://warewulf.lbl.gov/)

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.


vanessa s

unread,
Apr 2, 2017, 3:41:25 AM4/2/17
to singu...@lbl.gov
Hey Tin!

I haven't talked with Greg since this morning, but I think I have a bit of insight to some of the errors (which I've seen before):



Hope it is not something amiss with my config.  But while testing a build with Singularity 2.2.99, I am getting a number of warnings about missing user and group.  see below for details.


Host: RHEL 6.6
Container: Centos 7

When I ran that Singularity definition file on a system with the dev code for the upcoming Singularity 2.3, I saw some warnings about /etc/group and /etc/passwd.
I reran the same build using 2.2.1 and don't see them.

I think this is because the shell was just changed to bash for default, see here --> https://github.com/singularityware/singularity/issues/552 

In the resulting container build by "2.2.99", these number of system users in the passwd file were surprisingly slim:

Singularity topspin.4.img> cat /etc/passwd
systemd-bus-proxy:x:999:997:systemd Bus Proxy:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:998:996:User for polkitd:/:/sbin/nologin
tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin


 no idea about this one!
 

 > sudo -E singularity bootstrap topspin.img Singularity | tee sing.log 2>&1
Building from bootstrap definition recipe
Adding label: 'SINGULARITY_CONTAINER_UUID' = 'ecafe2ee-cc49-43c2-8869-9367d738a2e8'
Adding label: 'SINGULARITY_DEFFILE' = 'Singularity'
Adding label: 'SINGULARITY_BOOTSTRAP_VERSION' = '2.2.99'
Adding label: 'SINGULARITY_DEFFILE_BOOTSTRAP' = 'yum'
Adding label: 'SINGULARITY_DEFFILE_OSVERSION' = '7'
usage: add.py [-h] [--key KEY] [--value VALUE] [--file FILE] [-f]
add.py: error: unrecognized arguments: https
/e/libexec/singularity/bootstrap-scripts/pre.sh: line 64: basearch: unbound variable

This one I've definitely seen! The reason is because there is a bug that seems to prevent any kind of comment lines from being in the bootstrap spec - they get parsed as labels. You see the above because the labels are added to the container's metadata that takes two things, a --key and a --value to write to the labels json file in .singularity.d. What happens with the bug is that it reads the comment line with the https, splits it at the colon, and then passes it as an argument (https) and the optparse function doesn't know what to do with it, spits out the usage (first yellow line) and the error (second yellow line). I'm not sure about what is happening in pre, but possibly something wrong with MirrorURL because basearch is at the end of it. Methinks once the first error is resolved, this second one (parsing that line as  a label, period to lead to the unbound variable) will be resolved too.
 
Just a note for the first lines of post that (I think) are meant to be environment variables? You should put them in an %environment section for the new version. That might be why you see not enough users, if those variables are relevant for that.
I'm not good enough with this OS to know where in the bootstrap spec this step is happening. Could it be that extraction of some later files overwrites these, or more specifically, the entire /etc folder?
Best,

Vanessa
 
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
Reply all
Reply to author
Forward
0 new messages