Ansible container build error "Failed to get D-Bus connection: Operation not permitted" on Centos7

[pbsd...@gmail.com]

Hi, I'm creating an ansible container on Centos7 with 'system / systemctl' feature (enabled), but struggling with the above error.

I've gone through all internet posts and try various suggestions but no luck. Hope someone here can shine the light and help me out.

Here is the information about my environment and implement (note that modifications on confidential names/path/etc.)

docker info

---------------

Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 6
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-693.5.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 30.94 GiB
Name: f24-ll-02
ID: 7B3C:NDQN:S373:KEUV:7SHB:B2F4:4EH2:62OE:QOVA:SGTR:WI7T:DUVP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://xxxxxxx.com:80
Https Proxy: http://xxxxxxx.com:80
No Proxy: localhost,127.0.0.1
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

Docker version

-------------------

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Wed Mar  7 17:06:16 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Wed Mar  7 17:06:16 2018
 OS/Arch:         linux/amd64
 Experimental:    false

Ansible version

--------------------

ansible 2.5.0
  config file = /path/Desktop/DockerTest/vvsdfgs_test3/ansible.cfg
  configured module search path = [u'/home/4j36301/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]

container.yml (it's dirty - with all my trials commented or not)

-------------------------------------------------------------------------------

version: "2"
settings:

  conductor:
    # The Conductor container does the heavy lifting, and provides a portable
    # Python runtime for building your target containers. It should be derived
    # from the same distribution as you're building your target containers with.
    base: centos:7
    # roles_path:   # Specify a local path containing Ansible roles
    # volumes:      # Provide a list of volumes to mount
    environment:
      http_proxy: http://xxxxxxx.com:80
      https_proxy: http://xxxxxxx.com:80

  # Set the name of the project. Defaults to basename of the project directory.
  # For built services, concatenated with service name to form the built image name.
  project_name: New_vvvvvv_container

  # When using the k8s or openshift engines, use the following to set the namespace.
  # If not set, the project name will be used. For openshift, the namespace maps to a project,
  # and description and display_name are supported.
  k8s_namespace:
    name: some_name
    description: DevelopmentEnv
    display_name: some_name

  environment:
    container=docker

services:
  # Add your containers here, specifying the base image you want to build from.
  # To use this example, uncomment it and delete the curly braces after services key.
  # You may need to run `docker pull ubuntu:trusty` for this to work.
  gmx_install:
#    from: local/c7-systemd-httpd
#    from: centos:7
#    from: docker.io/centos/systemd
    from: centos/systemd
#    from: centos:centos7
    #from: rhel7:1.0

    cap_add:
      - SYS_ADMIN
#    security_opt:
#      - seccomp:unconfined

    roles:
      - yum_installs
      - some_installs

    tmpfs:
      - /run
    volumes:
      - /run:/tmp
      - /path/Desktop/DockerTest/test3/applogs:/applogs
      - /path/Desktop/DockerTest/test3/missiondata:/vvvvvdata
      - /path/Desktop/DockerTest/test3/xxxConfiguration:/xxxConfiguration
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
#    entrypoint: /bin/sh -c

#    entrypoint: /usr/sbin/init
    privileged: true
    become_user: root

    ports:
      - 161:161/udp
     ...
      - 16110:16110/udp

#    command: ["/usr/bin/dumb-init", "httpd", "-DFOREGROUND"]
    command:
    - /usr/local/bin/cmd.sh # cmd.sh contains "exec /usr/sbin/init"
    - /usr/lib/systemd/systemd
    - --system
#    command: ["/usr/sbin/init"]

Observations:

1) Build (sudo docker --debug build --no-cache) resulted the error in title.

2) It's weird that event the build failed with error, but it ended up with a running container, so I can "exec" into it. Tried some commands -

    a) 'ps faux' always displayed PID 1 with command 'sh -c while true sleep...' (while I expect '/usr/sbin/init'), not sure where the command is from, trying to change 'command' in container.yml makes no change.

   b) 'systemctl --version' does returned valid value.

   c) 'systemctl status' returned the error in title.

3) 'docker ps -a' listed the container images.

4) 'docker images' listed the conductor (...._container-conductor) image and the container image. Container image has "<none>" with 'REPOSITORY" and "TAG", its ID is valid and size is normal.

5) Before each build, I always stop and delete (docker stop ID & docker rm ID) the running container, and I also remove the "<none>" image, assuming that will allow me to conduct 'clean' new build. Not sure if that's enough or too much for a clean new build.

6) Ansible.cfg and requirements.yml are empty (default). 

7) In the 'roles', there is nothing weird, just copy files and install our product and config/start/stop/restart some services via 'systemd' 

8) Most of times, the build ended up with the same container name as the previous, I think that's because there is not enough layer changes so docker consider that as same container (image).

TIA

     

[Jim Robinson]

The problem your having is ansible-container can't complete the build because systemd isn't playing well with the restricted environment of the docker container.  When I was exploring this I came across the following article, which you might find helpful:

https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/

I don't know your context, but for my own project I concluded that it was better to not even try to use systemd based services at all within Docker.  It appears to be *possible* to do it (as outlined in the article), but I worried about it being a fragile workaround.

[pbsd...@gmail.com]

Thank you Jim for the information, I've confirmed your statement ('systemd' not available if the build is not complete WITHOUT errors).

Now I tweaked to make the build complete without errors, but have to strip out those ansible roles using 'systemd'. Is there a way to make the roles to skip/hide/avoid the 'build' and only execute them at (ansible-container) 'run' or (docker) 'exec' time?

[Jim Robinson]

If you explain a bit more about what one of your problematic roles looks like, what it is doing with systemd, people on the list might be able to offer suggestions.  It's not clear to me from your earlier example whether you're merely launching systemd as part of the build, executing handlers to restart services, or something else.

[pbsd...@gmail.com]

In 'container.yml' -

.......

roles:

  - my-role

.....

In 'my-role/tasks/main.yml'

.....

- name: task name

    shell: /path/to/myscript.sh

......

In 'myscript.sh'

.....

    /bin/systemctl start my-service

.....

As I posted earlier, I got D-Bus error in build time, on the ansible role (my-role) at 'task name' (executing 'shell: /path/to/myscript.sh'), even I made the task as the last one in 'main.yml' file. Based on your reply, that's because systemd wouldn't be available until the build completes successfully (without error).

Your explanation is proved that if I removed the task (executing 'shell: /path/to/myscript.sh'), the build completed without error, and then I can use 'systemctl' command after I login (exec) to the container (if the build failed with D-Bus error as before, I got the same error when using 'systemctl' after I login to the container). And if I now execute 'myuscript.sh' with the 'docker exec' command (i.e. 'docker exec ...... -c "/path/to/myscript.sh") it's now executed successfully.

So in my view, there are three ways to include 'myscript' -

• Build time (ideal): as described at the beginning of this reply, to include/use it inside main.yml file. Problem is that build will fail, so I can't
• Run time (good): the container is started with 'ansible-container run ....', I hope there is a way to pass in and execute 'myscript' in this command, but it seems there are limited options for 'ansible-container'. The only promising way is '--role-path', but according to this post, any roles inside the project will be applied to build/run/install/etc., which is not good to my case. I want the role (or 'myscript.sh') to be only executed in 'ansible-container run' command.
• Login/exec (not good): I now have to run 'extra' command ('docker exec ...... -c "/path/to/myscript.sh") after I start ('ansible-container run ...') the container to make 'myscript.sh' executed, that's not good but that's the only way since neither of the above two works. 

Hope this clarify a little bit about my question(s) and you can point out a better solution!

[Jim Robinson]

I think you'll find you have problems trying to run the final image even if you do manage to get it built unless you make changes to the image as described in that article.

To answer your specific question, since you are launching systemd from within a shell script I suppose I'd say you could do something like check the state of the system before you decide whether or not to actually launch the systemd process.  For example, if /_usr is actually mounted instead of empty, it probably means you're running the 'build' phase.

There may be environment variables you can check as well, perhaps running a debugging test where you execute 'env' from your script to dump out the environment would let you look for a variable you can test against to guard against launching during that build phase.