Ansible Builder destructive failure

[Kevin Knox]

The day has come. I have to have an EE to talk to network devices. I'm following Calvin Remsburg's tutorial at https://www.youtube.com/watch?v=mTllPoQQFjg. Around the 46 minute point, he builds the EE.

When build my EE, the RHEL 9 box on which I'm doing the build is ruined. After doing it twice, the box will only run for 30 seconds. Happily, I was working in a sandbox, so no harm done, but what do I do?

Before running ansible-builder I must install pip so I can install builder
> python -m ensurepip --upgrade
> python -m pip install ansible-builder

I create 3 files:

execution-environment.yml
<#
--- 
version: 1
dependencies: 
    galaxy: requirements.yml
    python: requirements.txt
additional_build_steps:
    prepend: |
        RUN pip3 install --upgrade pip setuptools
    append: 
        - RUN ls -la /etc
#>

requirements.yml
<#
collections:
  - juniper.device
#> 
requirements.txt 
<#
junos-eznc
jsnapy
jxmlease
#>


Then execute the build from a locally registered base image:
> ansible-builder build --tag kevinplaying --build-arg EE_BASE_IMAGE=quay-remote.rtfx.mycompanysc.com/ansible/awx-ee:latest -v 3

When I did it the first time I was being stopped by some firewalls. I got that fixed. After that, it runs happily along until:
...
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
+ cp /tmp/src/requirements.txt /output/requirements.txt
++ ls -1 '/output/wheels/*whl'
++ wc -l
+ '[' 0 -gt 0 ']'
++ get-extras-packages
+ /usr/bin/dnf clean all
44 files removed
+ rm -rf /var/cache/dnf /var/cache/yum
+ rm -rf /var/lib/dnf/history.sqlite /var/lib/dnf/history.sqlite-shm /var/lib/dnf/history.sqlite-wal
+ rm -rf /var/log/dnf.librepo.log /var/log/dnf.log /var/log/dnf.rpm.log /var/log/hawkey.log
+ rm -rf /tmp/venv

Remote side unexpectedly closed network connection

(The server closed the the connection, because it powered itself off completely.)

Is "dnf" a sign the process did not finish? Is it just trying to abandon ship cleanly? Any guesses what's dying? I'll attach the log files in a post shortly.

[Kevin Knox]

Really, the other logs are only partials of the one I'm attaching here. The next log shows the server going down sooner, but now the server even goes down before I do the first step. So, I've actually destroyed the server. The next log is only showing the net effects of that. The previous log is just the same log without the verbosity. 

There is some chance this log is actually another example of a premature death due to the damage done in the first run. The log from the first run, without the verbosity, is in the previous message. 

[Kevin Knox]

Hung hard with "docker" as the container-runtime. Maybe better than destructive crashing with podman serving that role. 

[Kevin Knox]

Maybe good news. After an hour or two, the process started moving again, and after another couple hours, it ended. In between, I lost network connection to the box, so I cannot see any of the output from the process. Sigh. And I don't know where to look for the completed image. Sigh. But I'll get all that tomorrow. Thank you. 

Still pending is the question why the process blew up my Linux box using podman, but if I can do this with a docker build, then I'm happy. 

[Kevin Knox]

The build failed. I have to figure out how to upgrade Python, I gather. 

#19 24.90 Complete!
#19 24.97 + '[' -f /output/bindep/run.txt ']'
#19 24.97 ++ cat /output/bindep/run.txt
#19 24.97 + PACKAGES='python3-rpm
#19 24.97 python38-cryptography
#19 24.97 python38-pytz
#19 24.97 python38-pyyaml
#19 24.97 python38-requests
#19 24.97 qemu-img
#19 24.97 rsync'
#19 24.97 + '[' '!' -z 'python3-rpm
#19 24.97 python38-cryptography
#19 24.97 python38-pytz
#19 24.97 python38-pyyaml
#19 24.97 python38-requests
#19 24.97 qemu-img
#19 24.97 rsync' ']'
#19 24.97 + /usr/bin/dnf install -y python3-rpm python38-cryptography python38-pytz python38-pyyaml python38-requests qemu-img rsync
#19 25.50 Last metadata expiration check: 0:00:06 ago on Wed Apr 12 14:04:14 2023.
#19 25.78 Package python3-rpm-4.16.1.3-22.el9.x86_64 is already installed.
#19 25.79 No match for argument: python38-cryptography
#19 25.80 No match for argument: python38-pytz
#19 25.81 No match for argument: python38-pyyaml
#19 25.82 No match for argument: python38-requests
#19 25.82 Package qemu-img-17:7.2.0-14.el9.x86_64 is already installed.
#19 25.82 Package rsync-3.2.3-19.el9.x86_64 is already installed.
#19 25.83 Error: Unable to find a match: python38-cryptography python38-pytz python38-pyyaml python38-requests
#19 ERROR: process "/bin/sh -c /output/install-from-bindep && rm -rf /output/wheels" did not complete successfully: exit code: 1
------
 > [stage-2 5/6] RUN /output/install-from-bindep && rm -rf /output/wheels:
#19 24.97 + /usr/bin/dnf install -y python3-rpm python38-cryptography python38-pytz python38-pyyaml python38-requests qemu-img rsync
#19 25.50 Last metadata expiration check: 0:00:06 ago on Wed Apr 12 14:04:14 2023.
#19 25.78 Package python3-rpm-4.16.1.3-22.el9.x86_64 is already installed.
#19 25.79 No match for argument: python38-cryptography
#19 25.80 No match for argument: python38-pytz
#19 25.81 No match for argument: python38-pyyaml
#19 25.82 No match for argument: python38-requests
#19 25.82 Package qemu-img-17:7.2.0-14.el9.x86_64 is already installed.
#19 25.82 Package rsync-3.2.3-19.el9.x86_64 is already installed.
#19 25.83 Error: Unable to find a match: python38-cryptography python38-pytz python38-pyyaml python38-requests
------
Dockerfile:30
--------------------
  28 |
  29 |     COPY --from=builder /output/ /output/
  30 | >>> RUN /output/install-from-bindep && rm -rf /output/wheels
  31 |     RUN ls -la /etc
  32 |     LABEL ansible-execution-environment=true
--------------------
ERROR: failed to solve: process "/bin/sh -c /output/install-from-bindep && rm -rf /output/wheels" did not complete successfully: exit code: 1

So, here's where my head is:

I blew my box up with podman, because I was running no container platform. I figure the process maybe needs a container manager running somewhere into which it builds the things. Without that, it ate the box in some way. I had to install docker to run ansible-builder with container-runtime of docker, so it built into that container platform. 

If that's right, then there should probably be some kind of idiot-check in ansible-builder to tell me I need a container platform running. 

Last night's process took a couple hours before my network went down and I did not see this final output. I'm going to guess it's taking less long now due to caching, but I'm probably seeing the same error now as I saw last night. So, off I go to learn how to admin python. 

The easier things get, the less I'm able to do them. 

Sigh. 

[Kevin Knox]

Ansible-builder wants python38-cryptography and other packages that don't exist. What am I supposed to do about this?

I installed my dependencies and installed cryptography successfully, but dnf is trying to install things that just seem not to exist?

yum install gcc libffi-devel python3-devel openssl-devel
python3 -m pip install cryptography

python3 -m pip install python38-cryptography python38-pytz python38-pyyaml python38-requests qemu-img rsync
ERROR: Could not find a version that satisfies the requirement python38-cryptography (from versions: none)
ERROR: No matching distribution found for python38-cryptography

[Kevin Knox]

Installing python38 packages seems to be impossible on RHEL 9 and hard on RHEL 8. 

>  dnf -y install python38-cryptography python38-pytz python38-pyyaml python38-requests qemu-img rsync

That works on RHEL 8 after yumming python38 etc. 

[Kevin Knox]

This is unplayable. I really need help.

I finally got ansible-builder to run to completion, and nothing is output.

I ran:
ansible-builder build --tag docker-local.rtfx.mycompanysc.com/ansible-ee-juniper:1.0 --build-arg ANSIBLE_GALAXY_CLI_COLLECTION_OPTS='--ignore-certs' --container-runtime docker > build_output_terse.log

My output was:
Running command:
  docker build -f context/Dockerfile -t docker-local.rtfx.mycompanysc.com/ansible-ee-juniper:1.0 --build-arg=ANSIBLE_GALAXY_CLI_COLLECTION_OPTS=--ignore-certs context
Complete! The build context can be found at: /root/context

This is my docker status:
[root@SelfServ30244 ~]# docker image ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
[root@SelfServ30244 ~]#

This is what's in my context folder:
[root@SelfServ30244 context]# ls -ltrR
.:
total 4
drwxr-x--- 2 root root   54 Apr 12 15:44 _build
-rw-r----- 1 root root 1177 Apr 13 14:55 Dockerfile

./_build:
total 8
-rw-r----- 1 root root 32 Apr 12 15:42 requirements.yml
-rw-r----- 1 root root 27 Apr 12 15:42 requirements.txt

It succeeded, and there's no artifact at all to indicate anything happened.

What do I do?

[Kevin Knox]

Zero image generated by a successful, completed run. I'm baffled. 

The command:

ansible-builder build --tag docker-local.rtfx.mycompanysc.com/ansible-ee-juniper:1.0 --build-arg ANSIBLE_GALAXY_CLI_COLLECTION_OPTS='--ignore-certs'  -v 3 --container-runtime docker > build_output_verbose.log

Ends with:

#21 0.216 drwxr-xr-x 1 root root     21 Mar 16  2022 xdg
#21 0.216 drwxr-xr-x 2 root root      6 Jun 21  2021 xinetd.d
#21 0.216 drwxr-xr-x 1 root root     57 Apr 13 20:12 yum
#21 0.216 lrwxrwxrwx 1 root root     12 Jan 12 09:05 yum.conf -> dnf/dnf.conf
#21 0.216 drwxr-xr-x 1 root root   4096 Apr 28  2022 yum.repos.d
#21 DONE 0.3s

Complete! The build context can be found at: /root/context

And the /root/context folder contains:

[root@SelfServ30244 ~]# ls -ltrR context
context:

total 4
drwxr-x--- 2 root root   54 Apr 12 15:44 _build

-rw-r----- 1 root root 1177 Apr 13 16:09 Dockerfile

context/_build:

total 8
-rw-r----- 1 root root 32 Apr 12 15:42 requirements.yml
-rw-r----- 1 root root 27 Apr 12 15:42 requirements.txt

[root@SelfServ30244 ~]#

And the docker images list:

[root@SelfServ30244 ~]# docker image ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
[root@SelfServ30244 ~]#

[Kevin Knox]

I made it through all this to yet another problem captured in another thread. I will post the answer to that thread here as well, just to let any who follow know it did all work:

Resolved: Happily. I was able to phone a friend. 

My blindspot was around bindep.txt (https://docs.openstack.org/infra/bindep/). My python dependencies were all resolved by the following bindep.txt, though I'm sure I can remove some of these and still succeed. 

findutils [compile platform:centos-8 platform:rhel-8]
gcc [compile platform:centos-8 platform:rhel-8]
make [compile platform:centos-8 platform:rhel-8]
python38-devel [compile platform:centos-8 platform:rhel-8]
python38-cffi [platform:centos-8 platform:rhel-8]
python38-cryptography [platform:centos-8 platform:rhel-8]
python38-pycparser [platform:centos-8 platform:rhel-8]