Curl error 60 (after MLM 5.1 upgrade)

[Alex Gooch]

Good morning!

Just for some quick background, we recently upgraded/migrated our SUMA environment to MLM5.1. As a part of that process, we generated new certs for the hub. I suspect that this is causing my issue, but I am not sure exactly how to resolve it.

This is an example of the errors that I am seeing during the build process:

After the package/collection stage:

[ INFO    ]: Processing: [                                        ] 0%

[ DEBUG   ]: 10:29:22 | system: Building repository 'Leap_15_6' cache [....done]
[ DEBUG   ]: 10:29:23 | system: Building repository 'Leap_15_6_backports' cache [....done]
[ DEBUG   ]: 10:29:37 | system: Building repository 'Leap_15_6_sle' cache [....done]
[ DEBUG   ]: 10:29:38 | system: Building repository 'Leap_15_6_update' cache [....done]
[ DEBUG   ]: 10:29:38 | system: Retrieving repository 'backports_15_6_updates' metadata [...error]
[ DEBUG   ]: 10:29:38 | system: Warning: Skipping repository 'backports_15_6_updates' because of the above error.
[ DEBUG   ]: 10:29:38 | system: Retrieving repository 'leap_15_6_pool' metadata [...error]
[ DEBUG   ]: 10:29:38 | system: Warning: Skipping repository 'leap_15_6_pool' because of the above error.
[ DEBUG   ]: 10:29:38 | system: Retrieving repository 'leap_15_6_updates' metadata [...error]
[ DEBUG   ]: 10:29:38 | system: Warning: Skipping repository 'leap_15_6_updates' because of the above error.
[ DEBUG   ]: 10:29:38 | system: Retrieving repository 'libnvidia' metadata [...error]
[ DEBUG   ]: 10:29:38 | system: Warning: Skipping repository 'libnvidia' because of the above error.
[ DEBUG   ]: 10:29:38 | system: Retrieving repository 'mysql_80' metadata [...error]
[ DEBUG   ]: 10:29:38 | system: Warning: Skipping repository 'mysql_80' because of the above error.

...

When it ultimately fails, I see errors like this:

[ ERROR   ]: 10:29:44 | KiwiInstallPhaseFailed: System package installation failed: Repository 'backports_15_6_updates' is invalid.
[backports_15_6_updates|http://some_server.company.com/rhn/manager/download/spe_leap_15_6_x86_64-production-opensuse-backports-15.6-updates-x86_64] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://some_server.company.com/rhn/manager/download/spe_leap_15_6_x86_64-production-opensuse-backports-15.6-updates-x86_64'
 - Download (curl) error for 'http://some_server.company.com  /rhn/manager/download/spe_leap_15_6_x86_64-production-opensuse-backports-15.6-updates-x86_64/content':
   Error code: Curl error 60
   Error message: SSL certificate problem: unable to get local issuer certificate

Please check if the URIs defined for this repository are pointing to a valid repository.
Repository 'leap_15_6_pool' is invalid.
[leap_15_6_pool|http://some_server.company.com  /rhn/manager/download/spe_leap_15_6_x86_64-production-opensuse-leap-15.6-pool-x86_64] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://some_server.company.com  /rhn/manager/download/spe_leap_15_6_x86_64-production-opensuse-leap-15.6-pool-x86_64'
 - Download (curl) error for 'http://some_server.company.com  /rhn/manager/download/spe_leap_15_6_x86_64-production-opensuse-leap-15.6-pool-x86_64/content':
   Error code: Curl error 60
   Error message: SSL certificate problem: unable to get local issuer certificate

...

At this point, my appliance.kiwi file is the same as it was before the MLM upgrade. I read through other posts and have confirmed that I already had the following in my build file:

    <packages type="bootstrap">

        <package name="ca-certificates"/>

        <package name="ca-certificates-cacert"/>

        <package name="ca-certificates-mozilla"/>

On my newly migrated hub (where I am sourcing the packages), I have these certificate files generated:

/systems/certs
└── /systems/certs/server
    ├── /systems/certs/server/certs
    │   ├── /systems/certs/server/certs/combined-server.pem
    │   ├── /systems/certs/server/certs/server.company.com.key
    │   └── /systems/certs/server/certs/server.company.com.pem
    ├── /systems/certs/server/trust
    │   └── /systems/certs/server/trust/anchors
    │       ├── /systems/certs/server/trust/anchors/company-ca-combined.pem
    │       ├── /systems/certs/server/trust/anchors/company-ca-t1.pem
    │       └── /systems/certs/server/trust/anchors/company-root-ca.pem
    ├── /systems/certs/server/server.company.com.cert.download.decoded
    ├── /systems/certs/server/server.company.com.cert.download.result
    ├── /systems/certs/server/server.company.com.cnf
    ├── /systems/certs/server/server.company.com.csr
    ├── /systems/certs/server/server.company.com.csr.check.result
    ├── /systems/certs/server/server.company.com.csr.enrollment.result
    └── /systems/certs/server/server.company.com.key

Do I need to install any of these files onto my kiwi server in order for the build to be able to reach the standalone hub? If so, could you point me to which ones I should install, and where on the kiwi build host? If this is not the issue, have you seen anything like this before? 

Any help is greatly appreciated!

Thank you,

  Alex

[Marcus Schäfer]

Hi Alex,

> Just for some quick background, we recently upgraded/migrated our SUMA
> environment to MLM5.1. As a part of that process, we generated new
> certs for the hub. I suspect that this is causing my issue, but I am
> not sure exactly how to resolve it.
>

> [ ERROR ]: 10:29:44 | KiwiInstallPhaseFailed: System package

> Error code: Curl error 60
> Error message: SSL certificate problem: unable to get local issuer

Yep you get this error if the respective certificates where not
installed and imported as part of your image build process.

I see the error to happen in the "system" phase. This means during
the bootstrap phase no certificates that allows certificate validation
against the repo server(s) you are using can be made. Usually you see
the following as part of the bootstrap section

<packages type="bootstrap">
<package name="ca-certificates"/>

<package name="ca-certificates-mozilla"/>
...

obviously this is not enough to meet the needs in you case.

So you need at best a certificate package containing the the certs
for the hub you are using. Alternatively you can also put them into
a tarball. Next to this add this into your bootstrap section

<package name="hub-certs"/>

or

<archive name="hub-cert.tar.gz"/>

You need to make sure the certs are placed in "/usr/share/pki/trust/anchors"
if you are using a self-signed cert you also need to make sure the
CA cert is placed there. Make sure to add the following to the
script: "post_bootstrap.sh"

update-ca-certificates

If you have the certs packaged, the above can also come from the
package %post

Hope this helps

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

[Alex Gooch]

Hi Marcus:

I tried what was suggested, but it doesnt look like there was any change from my side.

I packaged up my new certs:

 root@stand-alone-hub (10:05:38)  /tmp/hub-cert
 $ tar tzf /tmp/hub-cert.tar.gz
./
./usr/
./usr/share/
./usr/share/pki/
./usr/share/pki/trust/
./usr/share/pki/trust/anchors/
./usr/share/pki/trust/anchors/my-ca-combined.pem
./usr/share/pki/trust/anchors/my-ca-t1.pem
./usr/share/pki/trust/anchors/my-root-ca.pem

And I copied them over to my build server, on the same level as the appliance.kiwi. 

kiwi1:/home/me/git/spe-kiwi-qvr_gold/QVR_Gold_15_6 # ls -ltr
total 48
drwxr-xr-x 1 me me   186 May  8  2025 rpms
-rw-r--r-- 1  me me   14953 Feb 27 11:34 rhn-org-trusted-ssl-cert-1.0-3.noarch.rpm
drwxr-xr-x 1  me me     226 Feb 27 11:42 root
-rwxr-xr-x 1  me me    3646 Feb 27 14:43 config.sh
-rw-r--r-- 1  me me    3869 Mar 10 10:11 hub-cert.tar.gz
-rwxr-xr-x 1  me me   19166 Mar 10 10:16 appliance.kiwi
-rwxr-xr-x 1  me me    1305 Mar 10 10:18 post_bootstrap.sh

I updated the applience.kiwi file with the new file:

<packages type="bootstrap">
        <package name="ca-certificates"/>

        <package name="ca-certificates-cacert"/>
        <package name="ca-certificates-mozilla"/>
        <package name="cracklib-dict-full"/>
        <package name="dracut-kiwi-lib"/>
        <package name="filesystem"/>
        <package name="glibc-locale"/>
        <package name="http://stand-alone-hub.my.com/pub/rhn-org-trusted-ssl-cert-1.0-3.noarch.rpm"/>
        <archive name="hub-cert.tar.gz"/>

...

I also made sure to update post_bootstrap.sh. For transparency-sake, I was using post_bootstrap.sh for other things in a previous build. This is what that file looks like today:

#!/bin/bash
# Source keys
echo "Sourcing gpg keys..."
/usr/bin/wget http://prod-hub.my.com/pub/packagehub-gpg-pubkey-65176565.key -O /tmp/65176565.gpg
/usr/bin/wget http:// prod-hub.my.com  /pub/opensuse-gpg-pubkey-3dbdc284.key -O /tmp/3dbdc284.gpg
/usr/bin/wget http:// prod-hub.my.com  /pub/sle12-gpg-pubkey-39db7c82.key -O /tmp/39db7c82.gpg
/usr/bin/wget http:// prod-hub.my.com  /pub/sle11-gpg-pubkey-307e3d54.key -O /tmp/307e3d54.gpg
/usr/bin/wget http:// prod-hub.my.com  /pub/opensuse-gpg-pubkey-29b700a4.key -O /tmp/29b700a4.gpg
/usr/bin/wget http:// prod-hub.my.com  /pub/packagehub-gpg-pubkey-8A49EB0325DB7AE0.key -O /tmp/8A49EB0325DB7AE0.gpg
/usr/bin/wget http:// prod-hub.my.com  /pub/sle15-gpg-pubkey-3fa1d6ce.key -O /tmp/3fa1d6ce.gpg
mkdir -p /etc/pki/rpm-gpg

# Import keys
echo "Importing keys to local system..."
for gpg_file in $(find /tmp/*gpg); do
  key=$(echo $gpg_file | awk -F\/ '{print $NF}');
  echo "Copying ${gpg_file} to /etc/pki/rpm-gpg"
  /usr/bin/cp ${gpg_file} /etc/pki/rpm-gpg/${key}
  if [[ -e /etc/pki/rpm-gpg/${key} ]]; then echo "...complete!"; else echo "Copy of ${gpg_file} failed."; fi

  echo "Importing ${gpg_file}"
  /usr/bin/rpm --import ${gpg_file} && rm -rf ${gpg_file}
  echo $?
done

update-ca-certificates
exit

This is the log from that section of the build:

[ INFO    ]: 10:37:51 | Calling post_bootstrap.sh script
[ DEBUG   ]: 10:37:51 | --------------out start-------------
[ DEBUG   ]: 10:37:51 | Sourcing gpg keys...
[ DEBUG   ]: 10:37:51 | Importing keys to local system...
[ DEBUG   ]: 10:37:51 | Copying /tmp/29b700a4.gpg to /etc/pki/rpm-gpg
[ DEBUG   ]: 10:37:51 | ...complete!
[ DEBUG   ]: 10:37:51 | Importing /tmp/29b700a4.gpg
[ DEBUG   ]: 10:37:51 | 0
[ DEBUG   ]: 10:37:51 | Copying /tmp/307e3d54.gpg to /etc/pki/rpm-gpg
[ DEBUG   ]: 10:37:51 | ...complete!
[ DEBUG   ]: 10:37:51 | Importing /tmp/307e3d54.gpg
[ DEBUG   ]: 10:37:51 | 0
[ DEBUG   ]: 10:37:51 | Copying /tmp/39db7c82.gpg to /etc/pki/rpm-gpg
[ DEBUG   ]: 10:37:51 | ...complete!
[ DEBUG   ]: 10:37:51 | Importing /tmp/39db7c82.gpg
[ DEBUG   ]: 10:37:51 | 0
[ DEBUG   ]: 10:37:51 | Copying /tmp/3dbdc284.gpg to /etc/pki/rpm-gpg
[ DEBUG   ]: 10:37:51 | ...complete!
[ DEBUG   ]: 10:37:51 | Importing /tmp/3dbdc284.gpg
[ DEBUG   ]: 10:37:51 | 0
[ DEBUG   ]: 10:37:51 | Copying /tmp/3fa1d6ce.gpg to /etc/pki/rpm-gpg
[ DEBUG   ]: 10:37:51 | ...complete!
[ DEBUG   ]: 10:37:51 | Importing /tmp/3fa1d6ce.gpg
[ DEBUG   ]: 10:37:51 | 0
[ DEBUG   ]: 10:37:51 | Copying /tmp/65176565.gpg to /etc/pki/rpm-gpg
[ DEBUG   ]: 10:37:51 | ...complete!
[ DEBUG   ]: 10:37:51 | Importing /tmp/65176565.gpg
[ DEBUG   ]: 10:37:51 | 0
[ DEBUG   ]: 10:37:51 | Copying /tmp/8A49EB0325DB7AE0.gpg to /etc/pki/rpm-gpg
[ DEBUG   ]: 10:37:51 | ...complete!
[ DEBUG   ]: 10:37:51 | Importing /tmp/8A49EB0325DB7AE0.gpg
[ DEBUG   ]: 10:37:51 | 0
[ DEBUG   ]: 10:37:51 | --------------out stop--------------
[ DEBUG   ]: 10:37:51 | --------------err start-------------
[ DEBUG   ]: 10:37:51 |
Redirecting output to ‘wget-log’.
Redirecting output to ‘wget-log.1’.
Redirecting output to ‘wget-log.2’.
Redirecting output to ‘wget-log.3’.
Redirecting output to ‘wget-log.4’.
Redirecting output to ‘wget-log.5’.
Redirecting output to ‘wget-log.6’.
[ DEBUG   ]: 10:37:51 | --------------err stop--------------

But even with that script only containing update-ca-certificates, I still get an overall build failure. 

Is there something that I am doing wrong or anything else we should be looking toward?

Thank you,

  Alex

[Alex Gooch]

Hey Marcus! 

I wanted to update you that I got it working. The issue that I was having was the validity of the certs themselves. Once I pulled the proper certs, the build started working. Thank you again for your help!

[Marcus Schäfer]

Hey Alex,

> I wanted to update you that I got it working. The issue that I was
> having was the validity of the certs themselves. Once I pulled the
> proper certs, the build started working. Thank you again for your help!

Great news, happy you resolved it :)

Best regards,