Python Error First attempt at a fork build

[J. Mixer]

All,

I have jumped into the realm of contributing to the Kiwi imaging project. I have forked my first build and loaded it on my newly loaded openSUSE 15.4 build machine (actually a VM.)

My first attempt at tox -e devel produces the following error:

mixer@mixerlinux:~/kiwi> tox -e devel
devel create: /home/mixer/kiwi/.tox/3
devel installdeps: -r.virtualenv.dev-requirements.txt
devel develop-inst: /home/mixer/kiwi
devel installed: alabaster==0.7.12,anymarkup==0.8.1,anymarkup-core==0.8.1,attrs==21.4.0,Babel==2.10.3,bleach==4.1.0,bump2version==1.0.1,bumpversion==0.6.0,certifi==2022.6.15,cffi==1.15.1,charset-normalizer==2.0.12,click==8.0.4,colorama==0.4.5,configobj==5.0.6,coverage==6.2,cryptography==37.0.4,dataclasses==0.8,distlib==0.3.5,docopt==0.6.2,docutils==0.17.1,execnet==1.9.0,filelock==3.4.1,flake8==3.9.2,generateDS==2.29.24,ghp-import==2.1.0,idna==3.3,imagesize==1.4.1,importlib-metadata==4.8.3,importlib-resources==5.4.0,iniconfig==1.1.1,jeepney==0.7.1,Jinja2==3.0.3,json5==0.9.8,keyring==23.4.1,-e git+https://github.com/jbmixer/kiwi@7fdf127e89ac97b45e140b99112b059b9df5bb84#egg=kiwi,lxml==4.9.1,MarkupSafe==2.0.1,mccabe==0.6.1,mock==4.0.3,mypy==0.971,mypy-extensions==0.4.3,packaging==21.3,pkginfo==1.8.3,platformdirs==2.4.0,pluggy==1.0.0,py==1.11.0,pycodestyle==2.7.0,pycparser==2.21,pyenchant==3.2.2,pyflakes==2.3.1,Pygments==2.12.0,pyparsing==3.0.9,pytest==7.0.1,pytest-container==0.0.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-testinfra==6.8.0,pytest-xdist==2.5.0,python-dateutil==2.8.2,pytz==2022.1,pyxattr==0.7.2,PyYAML==6.0,readme-renderer==34.0,requests==2.27.1,requests-toolbelt==0.9.1,rfc3986==1.5.0,rstxml2docbook @ git+https://github.com/openSUSE/rstxml2docbook.git@f000fb64b79011e216a6981592fced2ef3c92c75,SecretStorage==3.3.2,simplejson==3.17.6,six==1.16.0,snowballstemmer==2.2.0,Sphinx==5.0.2,sphinx-rtd-theme==1.0.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,sphinxcontrib-spelling==7.6.0,toml==0.10.2,tomli==1.2.3,tox==3.25.1,tqdm==4.64.0,travis-sphinx==2.2.1,twine==3.8.0,typed-ast==1.5.4,types-mock==4.0.15,types-pkg-resources==0.1.3,types-PyYAML==6.0.10,types-requests==2.28.3,types-simplejson==3.17.7,types-urllib3==1.26.16,typing_extensions==4.1.1,urllib3==1.26.10,virtualenv==20.15.1,webencodings==0.5.1,xattr==0.9.3,xmltodict==0.13.0,zipp==3.6.0
devel runtests: PYTHONHASHSEED='2271144615'
devel runtests: commands[0] |
Traceback (most recent call last):
  File "/usr/bin/tox", line 11, in <module>
    load_entry_point('tox==2.9.1', 'console_scripts', 'tox')()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 40, in main
    retcode = Session(config).runcommand()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 392, in runcommand
    return self.subcommand_test()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 583, in subcommand_test
    self.runtestenv(venv)
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 592, in runtestenv
    self.hook.tox_runtest(venv=venv, redirect=redirect)
  File "/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/lib/python3.6/site-packages/tox/venv.py", line 464, in tox_runtest
    venv.test(redirect=redirect)
  File "/usr/lib/python3.6/site-packages/tox/venv.py", line 373, in test
    if argv[0].startswith("-"):
IndexError: list index out of range

My "rusty" Python knowledge tells me that the array in question is empty (or isn't an array.) Does anyone have a suggestion as to what is wrong with the build?

Regards,

J.

[J. Mixer]

All,

Additionally, I ran just "tox" after this failure. I received some initial errors during the "check" phase, but all of the tests then ran. Here is the summary output:

ERROR:   check: commands failed
SKIPPED:  unit_py3_10: InterpreterNotFound: python3.10
SKIPPED:  unit_py3_9: InterpreterNotFound: python3.9
SKIPPED:  unit_py3_8: InterpreterNotFound: python3.8
  unit_py3_6: commands succeeded
  packagedoc: commands succeeded

Regards,

J.

[Marcus Schäfer]

Hi,

sorry for the delay

> I have jumped into the realm of contributing to the Kiwi imaging

Thanks for doing this

> My first attempt at tox -e devel produces the following error:

Hmm, very strange it fails in tox itself. From where did you get
tox ? Is this installed as package from your 15.4 system ?

Try the following:

mixer@mixerlinux:~/kiwi> rm -rf .tox/3
mixer@mixerlinux:~/kiwi> sudo pip3 install --upgrade tox

mixer@mixerlinux:~/kiwi> tox -e devel

It completed for me without any issues.
I'm using tox:

tox --version
3.25.1 imported from /usr/lib/python3.6/site-packages/tox/__init__.py

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[J. Mixer]

Marcus,

It took me a couple of days to get back to this.

Hmm, very strange it fails in tox itself. From where did you get
tox ? Is this installed as package from your 15.4 system ?

Try the following:

mixer@mixerlinux:~/kiwi> rm -rf .tox/3
mixer@mixerlinux:~/kiwi> sudo pip3 install --upgrade tox
mixer@mixerlinux:~/kiwi> tox -e devel

pip install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  WARNING: The scripts pip, pip3 and pip3.6 are installed in '/home/mixer/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-21.3.1

It completed for me without any issues.

Mine did not, but it got further:

(excerpt from log output)

error: can't find Rust compiler

Then I executed:

mixer@mixerlinux:~/kiwi $ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

(Rust install output between the ======== lines:)

=================================================================

info: downloading installer

Welcome to Rust!

This will download and install the official compiler for the Rust
programming language, and its package manager, Cargo.

Rustup metadata and toolchains will be installed into the Rustup
home directory, located at:

  /home/mixer/.rustup

This can be modified with the RUSTUP_HOME environment variable.

The Cargo home directory is located at:

  /home/mixer/.cargo

This can be modified with the CARGO_HOME environment variable.

The cargo, rustc, rustup and other commands will be added to
Cargo's bin directory, located at:

  /home/mixer/.cargo/bin

This path will then be added to your PATH environment variable by
modifying the profile files located at:

  /home/mixer/.profile
  /home/mixer/.bashrc

You can uninstall at any time with rustup self uninstall and
these changes will be reverted.

Current installation options:


   default host triple: x86_64-unknown-linux-gnu
     default toolchain: stable (default)
               profile: default
  modify PATH variable: yes

1) Proceed with installation (default)
2) Customize installation
3) Cancel installation
>1

info: profile set to 'default'
info: default host triple is x86_64-unknown-linux-gnu
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2022-07-19, rust version 1.62.1 (e092d0b6b 2022-07-16)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 54.0 MiB /  54.0 MiB (100 %)  30.8 MiB/s in  2s ETA:  0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
 18.3 MiB /  18.3 MiB (100 %)   4.7 MiB/s in  3s ETA:  0s
info: installing component 'rust-std'
 26.1 MiB /  26.1 MiB (100 %)  11.9 MiB/s in  2s ETA:  0s
info: installing component 'rustc'
 54.0 MiB /  54.0 MiB (100 %)  13.4 MiB/s in  4s ETA:  0s
info: installing component 'rustfmt'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu installed - rustc 1.62.1 (e092d0b6b 2022-07-16)


Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source "$HOME/.cargo/env"

====================================================================================

I had a couple of bad "tox -e devel" runs after that stating it still couldn't find Rust. Then I actually read the Rust install output and put the darn path to rust and .cargo in my PATH!

Then I was missing the openssl-devel package in my 15.4 installation.

mixer@mixerlinux:~ $ sudo zypper in openssl-devel

After that:

mixer@mixerlinux:~/kiwi $ tox -e devel

devel create: /home/mixer/kiwi/.tox/3
devel installdeps: -r.virtualenv.dev-requirements.txt
devel develop-inst: /home/mixer/kiwi

devel installed: alabaster==0.7.12,anymarkup==0.8.1,anymarkup-core==0.8.1,attrs==22.1.0,Babel==2.10.3,bleach==4.1.0,bump2version==1.0.1,bumpversion==0.6.0,certifi==2022.6.15,cffi==1.15.1,charset-

.

.

.
devel run-test-pre: PYTHONHASHSEED='3990406699'
_______________________________________________________________ summary _______________________________________________________________
  devel: commands succeeded
  congratulations :)

Now onto actually making some doc changes (and maybe some code changes too...)

Thanks again!

J.

[Marcus Schäfer]

Hi J,

> Mine did not, but it got further:

ok

> couldn't find Rust. Then I actually read the Rust install output and
> put the darn path to rust and .cargo in my PATH!

I really don't understand where the rust requirement should come from

> Then I was missing the openssl-devel package in my 15.4 installation.
> mixer@mixerlinux:~ $ sudo zypper in openssl-devel
> After that:
> mixer@mixerlinux:~/kiwi $ tox -e devel
> devel create: /home/mixer/kiwi/.tox/3
> devel installdeps: -r.virtualenv.dev-requirements.txt
> devel develop-inst: /home/mixer/kiwi
> devel installed:
> alabaster==0.7.12,anymarkup==0.8.1,anymarkup-core==0.8.1,attrs==22.1.0,
> Babel==2.10.3,bleach==4.1.0,bump2version==1.0.1,bumpversion==0.6.0,cert
> ifi==2022.6.15,cffi==1.15.1,charset-
> .
> .
> .

> devel: commands succeeded
> congratulations :)

yep that is how it should look like. Usually the tox setup is
straight forward but there is one problem with tox. It does resolve
nicely within the python module requirements but if a python module
itself has dependencies to non-python libraries for example libopenssl
or some C library or any other component that is not python, tox
will not be able to resolved this (how should it) and then the install
of that python module failed. Even more of a problem is, if a python
module has some external object reference and can't find it, it tries
to compile it themselve. This means all the headers and dev-libs
becomes a requirement as well. To make this a bit easier for users
we provide a helper script that installs the non python requirements
distinguished by distro here:

https://github.com/OSInside/kiwi/blob/master/helper/install_devel_packages.sh

We also document that step in the contributing chapter here:

https://osinside.github.io/kiwi/contributing.html

In your case it seems we are missing some stuff ?

> Now onto actually making some doc changes (and maybe some code changes
> too...)

Sounds great, thanks in advance

[J. Mixer]

Marcus,

Thanks for the input!

To make this a bit easier for users
we provide a helper script that installs the non python requirements
distinguished by distro here:

https://github.com/OSInside/kiwi/blob/master/helper/install_devel_packages.sh

We also document that step in the contributing chapter here:

https://osinside.github.io/kiwi/contributing.html

In your case it seems we are missing some stuff ?

I ran the install_devel_packages.sh script before I started this. Something was just not right with my initial 15.4 package list.

I have made my initial change to the doc/source/concept_and_workflow/customize_the_boot_process.rst file and now have to figure out how to get the branch into my fork and issue the pull request.  I need to read more on git and GitHub. (Old guy needs to learn new tricks.)

Regards,

J.

[J. Mixer]

All,

I have made my first "pull request" for the tools_for_hook_script branch in my fork (uid: jbmixer).  I hope I did things correctly.  Please let me know.

Regards,

J.

[Marcus Schäfer]

Hi J,

I have added you as collaborator to the kiwi git. This means you
don't have to create PRs from a fork anymore but can create
branches from the upstream repo and pull requests from your
branches.

This imho simplifies the process on your end and with your
first PR on the doc side you already showed that you know
what you are doing :)

Thanks for any future effort you will do on the project