boxbuild: KiwiUriOpenError Name or service not known

[Jan Robinson]

Hello All

On trying a boxbuild we are getting an error where a repo is not found from "inside" a temporary running image.

A set up is probably needed.

If you could assist please.

Kiwi server Sles15.3

kiwi-tools-9.24.44-1 and so is the rest at: 9.24.44-1

Updated today from 9.24.43 result is the same.

kiwi-boxed-plugin==0.2.23 pip installed.

Descriptions: https://github.com/OSInside/kiwi-descriptions.git

Command:

kiwi-ng --debug --type iso system boxbuild --box leap --   \

  --description /global/kiwi-build-descriptions/suse/x86_64/suse-leap-15.3  \

   --set-repo obs://openSUSE:Leap:15.3/standard  \

     --target-dir /global/local/box-result

Error:
INFO: 14:05:06 | Setting up repository obs://openSUSE:Leap:15.3/standard
INFO: 14:05:06 | --> Type: rpm-md
INFO: 14:05:06 | --> Priority: 1
ERROR: 14:05:06 | KiwiUriOpenError: http://download.opensuse.org/repositories/openSUSE:/Leap:/15.3/standard: HTTPConnectionPool(host='download.opensuse.org', port=80): Max retries exceeded with url: /repositories/openSUSE:/Leap:/15.3/standard (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5dd4a5ae10>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Logs attached.

Thanks so much,

Jan

[Marcus Schäfer]

Hi Jan,

> On trying a boxbuild we are getting an error where a repo is not found
> from "inside" a temporary running image.

Thanks for giving it a try

> Kiwi server Sles15.3
> kiwi-tools-9.24.44-1 and so is the rest at: 9.24.44-1
> Updated today from 9.24.43 result is the same.
> kiwi-boxed-plugin==0.2.23 pip installed.
>
> Descriptions: https://github.com/OSInside/kiwi-descriptions.git
>
> Command:
>
> kiwi-ng --debug --type iso system boxbuild --box leap -- \
>
> --description
> /global/kiwi-build-descriptions/suse/x86_64/suse-leap-15.3 \
>
> --set-repo obs://openSUSE:Leap:15.3/standard \
>
> --target-dir /global/local/box-result

ok I called the exact same command on my system and that worked.
I assume a network connectivity issue. You can recall the above
with --box-debug

kiwi-ng --debug --type iso system boxbuild --box leap --box-debug -- ...

This will not shutdown the box and you can check from inside why
http://download.opensuse.org/repositories/openSUSE:/Leap:/15.3/standard
is not reachable.

The way the traffic is passed between guest and host is by using
virtio-net-pci. In you process table you should find something similar
to:

ps ax | grep qemu
21512 pts/6 Sl+ 1:43 /usr/bin/qemu-system-x86_64 -m 8096 -accel accel=kvm -cpu host -nographic -nodefaults -snapshot -kernel /home/ms/.kiwi_boxes/leap/kernel.x86_64 -append console=hvc0 root=/dev/vda1 rd.plymouth=0 kiwi="--type iso system build --set-repo obs://openSUSE:Leap:15.3/standard" sharing_backend=_9p_ -drive file=/home/ms/.kiwi_boxes/leap/Leap-Box.x86_64-1.15.3-System-BuildBox.qcow2,if=virtio,driver=qcow2,cache=off,snapshot=on -netdev user,id=user0,hostfwd=tcp::10022-:22 -device virtio-net-pci,netdev=user0 -device virtio-serial -chardev stdio,id=virtiocon0 -device virtconsole,chardev=virtiocon0 -fsdev local,security_model=mapped,id=fsdev0,path=/home/ms/Project/kiwi-descriptions/suse/x86_64/suse-leap-15.3 -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=kiwidescription -fsdev local,security_model=mapped,id=fsdev1,path=/tmp/mytest -device virtio-9p-pci,id=fs1,fsdev=fsdev1,mount_tag=kiwibundle -initrd /home/ms/.kiwi_boxes/leap/initrd.x86_64 -smp 4

maybe your host connects through a proxy or something similar to
the network and that prevents the name resolution to work properly
inside of the box

Hope this helps

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[Jan Robinson]

Hello Marcus

Thank you for the reply.

 

maybe your host connects through a proxy or something similar to
the network and that prevents the name resolution to work properly
inside of the box

It is as you mentioned we are using a proxy.

Setting up a zypper repo on our local network in the box works.

But setting https_proxy= http_proxy= inside the box does not refresh the remote repo

obs://openSUSE:Leap:15.3/standard in the box.

 

Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org

Are there a way round this? (passing proxy credentials to the box)

Thanks again,

Jan

[Marcus Schäfer]

Hi,

> It is as you mentioned we are using a proxy.

Ah ok

> Error code: Connection failed
> Error message: Could not resolve host: download.opensuse.org
> Are there a way round this? (passing proxy credentials to the box)

If you create a directory called:

boxroot/...

as part of your image description (The directory which also contains your
kiwi XML file and other data)

You can create a set of files which gets copied into the box VM
rootfs prior the kiwi call. It imho should be possible to provide
the needed proxy configuration as boxroot/... files.

Cheers,

[Jan Robinson]

Hi Marcus

A few things were tried.
Nothing worked so far.

boxroot created in --descripton directory with files etc/environment and etc/sysconfig/proxy

1.
Having added the proxy and user/password information in /etc/sysconfig/proxy
Allowed for: at the debug shell add an OBS repo and use it.
The kiwi build still failed with KiwiUriOpenError.

localhost:~ # env |grep -i proxy ( showed no proxy variables, yet repo worked)

This failed directly after adding the repo successfully :
kiwi-ng --logfile /bundle/result.log --type iso system build --set-repo obs://openSUSE:Leap:15.3/standard --description /description --target-dir /result


2. Next:
Typing exit at the debug prompt.

localhost:~ # exit

re-logged in to a prompt and doing an env there showed the proxy settings.

localhost:~ # env | grep -i proxy ( proxy settings present )

/etc/profile.d/profile.sh seems to have executed and set the variables.

Doing the build manually now worked. (proxy variables set)

localhost:~ # kiwi-ng --logfile /bundle/result.log --type iso system build --set-repo obs://openSUSE:Leap:15.3/standard --description /description --target-dir /result

....
[ INFO    ]: 07:38:12 | Result files:
[ INFO    ]: 07:38:12 | --> image_changes: /result/Leap-15.3_appliance.x86_64-1.15.3.changes
[ INFO    ]: 07:38:12 | --> image_packages: /result/Leap-15.3_appliance.x86_64-1.15.3.packages
[ INFO    ]: 07:38:12 | --> image_verified: /result/Leap-15.3_appliance.x86_64-1.15.3.verified
[ INFO    ]: 07:38:12 | --> live_image: /result/Leap-15.3_appliance.x86_64-1.15.3.iso
[ INFO    ]: 07:38:12 | Cleaning up BootImageDracut instance


3.
It seems that the "automatic login" does not 'load' the proxy settings from
either /etc/environment or /etc/sysconfig/proxy for the build in the qemu image.

[  OK  ] Started wicked network nanny service.
         Starting wicked managed network interfaces...
[    6.157716] No iBFT detected.
[    6.181122] NET: Registered protocol family 17

localhost login: root (automatic login)

Have a lot of fun...
localhost:~ # + declare custom_mount=
+ declare kiwi_version=
+ trap finish EXIT
.
.

Regards,
Jan

[Marcus Schäfer]

Hi,

> boxroot created in --descripton directory with files etc/environment
> and etc/sysconfig/proxy

ok

> 1.
> Having added the proxy and user/password information in
> /etc/sysconfig/proxy

good

> Allowed for: at the debug shell add an OBS repo and use it.
> The kiwi build still failed with KiwiUriOpenError.

I would assume if you set:

export HTTP_PROXY="http://myusername:mypas...@myproxy.example.com:8080"

and then call

run_kiwi

zypper should consume the proxy setting ?

The run_kiwi command is the script which calls kiwi inside of
the box

Does this change anything ?

Thanks

[Jan Robinson]

Hello Marcus

If dumped in the debug shell,  HTTP_PROXY=, is not set.

As mentioned run_kiwi fails.

If  HTTP_PROXY= is set manually run_kiwi works.

Or by exiting the debug shell, that just spawn another shell and that shell will have

HTTP_PROXY= set because of the login that reads /etc/sysconfig/proxy and run_kiwi works.

But of cause the iso is not available on the server unless you copy it from the container to the 

local machine manually in this debug mode.

So the run_kiwi is not called with the proxy settings applied.

This is what we had before, nothing changed.

Thanks so much.

Regards,

Jan

[Marcus Schäfer]

Hi Jan,

> If dumped in the debug shell, HTTP_PROXY=, is not set.
>
> As mentioned run_kiwi fails.
>
> If HTTP_PROXY= is set manually run_kiwi works.

Ah ok good

> Or by exiting the debug shell, that just spawn another shell and that
> shell will have
>
> HTTP_PROXY= set because of the login that reads /etc/sysconfig/proxy
> and run_kiwi works.
>
> But of cause the iso is not available on the server unless you copy it
> from the container to the
>
> local machine manually in this debug mode.
>
> So the run_kiwi is not called with the proxy settings applied.

yep I suggest we fix the kiwi plugin to be able to work with a proxy

I hope I find a spare cycle to add it soon

Regards,

[Marcus Schäfer]

Hi Jan,

> If HTTP_PROXY= is set manually run_kiwi works.

For the reference:

https://github.com/OSInside/kiwi-boxed-plugin/issues/51

Regards,

[Jan Robinson]

Hello Marcus

Just saw the build files are in result.

But of cause the iso is not available on the server unless you copy it from the container to the 

local machine manually in this debug mode.

Does this change anything ?

So yes although we had to set proxy manually, the rebuild files are present.

Sorry for that.

Jan. 

[Jan Robinson]

Hi Marcus,

Thank you for:

   https://github.com/OSInside/kiwi-boxed-plugin/issues/51

Appreciated.

Jan

[Marcus Schäfer]

Hi Jan,

I created a patch for for proxy issue:

https://github.com/OSInside/kiwi-boxed-plugin/pull/52

The idea is that we support reading profile file inside
of the box named /etc/boxprofile

You can then add boxroot/etc/boxprofile and put something
like:

export HTTP_PROXY="..."

or whatever else shell code, functions you need

Would be great if you can review the open PR

If this matches what you need I can create a new release.
As the change only affects what happens when the box starts
there will be new boxes which gets auto-fetched once you
redo boxbuild

Thoughts ?

Regards,

[Jan Robinson]

Hello Marcus

This seem like a good idea.

It did not activate the content: KiwiUriOpenError

Adding it manually works:

localhost:~ # source  /etc/boxprofile

localhost:~ # run_kiwi   <- is then OK

----------------------------------------------------------

+ '[' -d /description/boxroot ']'
+ rsync -zav /description/boxroot/ /
sending incremental file list
./
etc/
etc/boxprofile
etc/environment
etc/profile.d/
etc/sysconfig/
etc/sysconfig/proxy
etc/sysconfig/proxy.kiwi
root/

sent 914 bytes  received 115 bytes  2,058.00 bytes/sec
total size is 762  speedup is 0.74
+ mkdir -p /root/.gnupg
+ chmod 700 /root/.gnupg
+ run_build
+ local options
+ local logfile=/bundle/result.log
+ local exit_code_file=/bundle/result.code
+ type setenforce
+ echo 1
+ rm -rf /result
++ cut -f2 '-d"' /proc/cmdline
------------------------------------------------------

Thank you,

Jan