Having trouble with scikit-build

[Daniel Alley]

Hello,

I have been attempting to use scikit-build to make a nontrivial C library (which has a set of Python extensions) pip-installable.  The issues I've been having are that I can find configurations that will work for bdist but will break when using sdist, or configurations that will halfway work for both in a virtualenv but install everything in the wrong places and can't be installed on the system, or a configuration that works great for both sdist and bdist but breaks the existing CMake non-package-based installation.

I'm not sure if I'm doing something wrong and there's a really simple solution, or if I'm looking at a bug, or if the CMake for the library needs to be improved generally.

This is my work-in-progress branch: https://github.com/dralley/createrepo_c/tree/python-pkg

The first commit "works" for both sdist and bdist but I noticed that it's installing things in the wrong place.  Instead of installing it in </venv_dir/lib/python3.6/site-packages/createrepo_c/, it tries to install it <venv_dir>/python/.  This actually "works" but only in a virtualenv.

The second commit I moved the __init__.py out to a separate folder at the top level.  This works "correctly" for both sdist and bdist, but now the normal CMake install is broken and I'm not really sure how to resolve it (my CMake knowledge is not great).

Is there something simple that I'm doing wrong?  Or something that, generally speaking, could be done much simpler?

Thank you,

Daniel

[Daniel Alley]

I should clarify that when I say "first commit" etc. I meant, comparing my branch against the upstream: https://github.com/rpm-software-management/createrepo_c/compare/master...dralley:python-pkg

[Jean-Christophe Fillion-Robin]

Hi Daniel,

Thanks for reaching out.

I will have a look shortly. Also worth noting that the upcoming release of scikit-build addresses few issues. 

We also plan on adding a gallery of examples very soon.

Thanks

Jc

--
You received this message because you are subscribed to the Google Groups "scikit-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scikit-build...@googlegroups.com.
To post to this group, send email to scikit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scikit-build/455ca8b5-5f9d-4581-8c67-15187d72a7c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Alley]

Jean,

I've tried again with the new release (actually, with current git:master) and still have the same issues.  When building as a bdist_wheel, all is well, but when building as an sdist it fails with  "error: package directory 'createrepo_c' does not exist".  I've tried a bunch of things to try to get it to work, but I'm stuck.

https://github.com/rpm-software-management/createrepo_c/compare/master...dralley:python-pkg

[Jean-Christophe Fillion-Robin]

Hi Daniel,

Thanks for reaching out.

We will have a look and get back to you.

Jc

To view this discussion on the web visit https://groups.google.com/d/msgid/scikit-build/7db6beda-6389-4bf7-8d53-957bda0c50ec%40googlegroups.com.

[Jean-Christophe Fillion-Robin]

Hi Daniel,

Following up on this, using master version of scikit-build, I was able to create a sdist following these steps:

cd /tmp && mkdir scratch && cd $_

mkvirtualenv -p python3.5 createrepo_c-py35

git clone  https://github.com/dralley/createrepo_c -b python-pkg

cd createrepo_c

pip install scikit-build

python setup.py sdist

I am now looking at creating the wheel and will follow up shortly.

Thanks for your patience,

Jc

To unsubscribe from this group and stop receiving emails from it, send an email to scikit-build+unsubscribe@googlegroups.com.

To post to this group, send email to scikit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scikit-build/7db6beda-6389-4bf7-8d53-957bda0c50ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

+1 919 869 8849

[Jean-Christophe Fillion-Robin]

After installing the dependencies, I was able to create the wheel.

That said, it failed to run through auditwheel:

$ auditwheel repair ./dist/createrepo_c-0.10.0-cp35-cp35m-linux_x86_64.whl -w ./dist/

Repairing createrepo_c-0.10.0-cp35-cp35m-linux_x86_64.whl
Traceback (most recent call last):
  File "/home/jcfr/.virtualenvs/scikit-build-35/bin/auditwheel", line 11, in <module>
    sys.exit(main())
  File "/home/jcfr/.virtualenvs/scikit-build-35/lib/python3.5/site-packages/auditwheel/main.py", line 49, in main
    rval = args.func(args, p)
  File "/home/jcfr/.virtualenvs/scikit-build-35/lib/python3.5/site-packages/auditwheel/main_repair.py", line 77, in execute
    update_tags=args.UPDATE_TAGS)
  File "/home/jcfr/.virtualenvs/scikit-build-35/lib/python3.5/site-packages/auditwheel/repair.py", line 77, in repair_wheel
    soname)
ValueError: Cannot repair wheel, because required library "libcreaterepo_c.so.0" could not be located

[Jean-Christophe Fillion-Robin]

Hi Daniel,

Here is a topic improving the build-system of createrepo_c:

https://github.com/dralley/createrepo_c/pull/1

(1) With this changes, i was able to build the wheel on a modern linux distribution (e.g Ubuntu 15.04). But auditwheel reported the following:

$ auditwheel repair ./dist/createrepo_c-0.10.0-cp35-cp35m-linux_x86_64.whl -w ./dist/
Repairing createrepo_c-0.10.0-cp35-cp35m-linux_x86_64.whl

usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: cannot repair "./dist/createrepo_c-0.10.0-cp35-cp35m-linux_x86_64.whl" to "manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

(2) The next step was to try to build the wheel using manylinux1 but it failed because the glibs provided by Centos5 is too hold, and createrepo_c seems to require Glibc 2.16 providing the checksum functionality (based on the build error encountered so far)

To reproduce the build error on manylinux

------------------------------------------------------------

Here is how it can be done using the convenience manylinux images from dockcross (https://github.com/dockcross/dockcross#readme)

docker pull dockcross/manylinux-x64

docker run --rm dockcross/manylinux-x64 > ~/bin/dockcross-manylinux-x64
chmod +x ~/bin/dockcross-manylinux-x64

cd /tmp && mkdir scratch && cd $_

git clone  https://github.com/jcfr/createrepo_c -b python-pkg-jcfr

cd python-pkg-jcfr

# This will mount the current directory as /work in the docker image

~/bin/dockcross-manylinux-x64 bash

export PATH=/opt/python/cp35-cp35m/bin/:$PATH

sudo yum install -y bzip2-devel libxml2-devel xz-devel rpm-devel
pip install nose

CFLAGS=-fdiagnostics-color=always python setup.py bdist_wheel -- -DOPENSSL_ROOT_DIR:PATH=/usr/local/ssl -- -j1

In file included from ../../../src/mergerepo_c.c:42:0:
../../../src/cleanup.h: In function ‘cr_local_array_unref’:
../../../src/cleanup.h:63:1: warning: implicit declaration of function ‘g_array_unref’ [-Wimplicit-function-declaration]
 CR_DEFINE_CLEANUP_FUNCTION0(GArray*, cr_local_array_unref, g_array_unref)
 ^
../../../src/cleanup.h: In function ‘cr_local_checksum_free’:
../../../src/cleanup.h:64:29: error: ‘GChecksum’ undeclared (first use in this function)
 CR_DEFINE_CLEANUP_FUNCTION0(GChecksum*, cr_local_checksum_free, g_checksum_free)
                             ^

Can we build a newer glibc statically on manylinux1 ?

------------------------------------------------------------------------------

I gave a try ... but the linux headers version on Centos 5 are too hold.

~/bin/dockcross-manylinux-x64 bash

cd /work

glibc_version=2.16.0

curl -LO#  https://ftp.gnu.org/gnu/glibc/glibc-${glibc_version}.tar.gz
tar -xzvf glibc-${glibc_version}.tar.gz
mkdir glibc-${glibc_version}-build
mkdir glibc-${glibc_version}-install
cd glibc-${glibc_version}-build
../glibc-${glibc_version}/configure --prefix=/work/glibc-${glibc_version}-install

[...]

checking installed Linux kernel header files... TOO OLD!
configure: error: GNU libc requires kernel header files from
Linux 2.6.19 or later to be installed before configuring.
The kernel header files are found usually in /usr/include/asm and
/usr/include/linux; make sure these directories use files from
Linux 2.6.19 or later.  This check uses <linux/version.h>, so
make sure that file was built correctly when installing the kernel header
files.  To use kernel headers not from /usr/include/linux, use the
configure option --with-headers.

Will manylinux 2010 solve the problem ?

-----------------------------------------------------------

Since the new manylinux images (manylinux2010) based on Centos6 [1][2] that will soon be available also expect code to rely on glibc <= 2.12, this will not address our problem.

[1] https://github.com/pypa/manylinux/pull/182

[2] https://www.python.org/dev/peps/pep-0571/

How to move forward ?

----------------------------------

Option 1:

Update the code to comply with Centos5 requirements

Option 2:

Or we build the wheel using Centos 7. It won't be possible to upload the wheel on PyPI but we could still have them uploaded as GitHub release. This would allow user to install them doing:

python -m pip install itk --no-index \
  -f https://github.com/rpm-software-management/createrepo_c/releases/tag/latest

To build the wheel:

Image "dockbuild/centos7-devtoolset4-gcc5:latest" from provided  by dockbuild (https://github.com/dockbuild/dockbuild#readme) should do it.

Let's give a try:

cd /tmp && mkdir scratch && cd $_

git clone  https://github.com/jcfr/createrepo_c -b python-pkg-jcfr

cd python-pkg-jcfr

docker pull dockbuild/centos7-devtoolset4-gcc5:latest

docker run --rm dockbuild/centos7-devtoolset4-gcc5 > ~/bin/dockbuild-centos7-devtoolset4-gcc5
chmod +x ~/bin/dockbuild-centos7-devtoolset4-gcc5

dockbuild-centos7-devtoolset4-gcc5 bash

sudo yum install -y bzip2-devel libxml2-devel xz-devel rpm-devel expat-devel file-devel glib2-devel   rpm-devel  libcurl-devel sqlite-devel  zlib-devel

python -m venv ./venv

source ./venv/bin/activate

python -m pip install --upgrade pip

pip install scikit-build

pip install nose

CFLAGS=-fdiagnostics-color=always python setup.py bdist_wheel -- -DOPENSSL_ROOT_DIR:PATH=/usr/local/ssl -- -j1

Et voila,

Let me know if you have any questions,

Thanks

Jc

[Daniel Alley]

Jean,

Thank you so much for looking into this, I really appreciate it.

With respect to sdist problems, I worded my previous email poorly - when I said that I had issues with building the sdist, I actually meant that while the actual process of "python setup.py sdist" works fine, the resulting package is broken and will not build at installation time.  I tried it again just now and it remains true.

(env) [vagrant@pulp3 createrepo_c]$ pip install dist/createrepo_c-0.10.0.tar.gz
Processing ./dist/createrepo_c-0.10.0.tar.gz
  Installing build dependencies ... done
    Complete output from command python setup.py egg_info:
    running egg_info
    creating /tmp/pip-req-build-2t5y01z9/pip-egg-info/createrepo_c.egg-info
    writing /tmp/pip-req-build-2t5y01z9/pip-egg-info/createrepo_c.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-req-build-2t5y01z9/pip-egg-info/createrepo_c.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-req-build-2t5y01z9/pip-egg-info/createrepo_c.egg-info/top_level.txt
    writing manifest file '/tmp/pip-req-build-2t5y01z9/pip-egg-info/createrepo_c.egg-info/SOURCES.txt'

    error: package directory 'createrepo_c' does not exist

I believe the issue is that pip is looking for a package named "createrepo_c" and finding that it doesn't exist, because cmake has not yet created that location and installed the __init__.py and the compiled library there.

https://github.com/dralley/createrepo_c/blob/8d954395551342324c5f72c29cbdc843b36ff72c/src/python/CMakeLists.txt#L67

If I change the "package_dir" in setup.py to something like "src/python", which does exist, it will build the package - but the result is unusable for bdist and sdist both.

(env) [vagrant@pulp3 createrepo_c]$ pip install dist/createrepo_c-0.10.0.tar.gz
Processing ./dist/createrepo_c-0.10.0.tar.gz
  Installing build dependencies ... done
Building wheels for collected packages: createrepo-c
  Running setup.py bdist_wheel for createrepo-c ... done
  Stored in directory: /home/vagrant/.cache/pip/wheels/c6/ff/12/5e571eb121e780436f5aafa3be199d7ad833d4a923fcf1e89d
Successfully built createrepo-c
Installing collected packages: createrepo-c
Successfully installed createrepo-c-0.10.0
(env) [vagrant@pulp3 createrepo_c]$

(env) [vagrant@pulp3 createrepo_c]$ ipython3

Python 3.6.5 (default, Mar 29 2018, 18:20:46)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.
 

In [1]: import createrepo_c
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-92f66d993759> in <module>()
----> 1 import createrepo_c

/usr/local/lib64/python3.6/site-packages/createrepo_c/__init__.py in <module>()
      2 """
      3
----> 4 from . import _createrepo_c
      5 from ._createrepo_c import *
      6

ImportError: cannot import name '_createrepo_c'
 

​With respect to manylinux, since this is my first foray into packaging a compiled Python package like this and I've never dealt with manylinux before, I don't really understand all the complexities here, at least not on a first reading.  I will reproduce the steps you have provided here and do some research on my own to understand this issue better.  I believe PyPI and Centos 6 support would both be hard requirements.

Thank you again for all your help, I really appreciate it.

Daniel

 

[Daniel Alley]

(2) The next step was to try to build the wheel using manylinux1 but it failed because the glibs provided by Centos5 is too hold, and createrepo_c seems to require Glibc 2.16 providing the checksum functionality (based on the build error encountered so far)

​Actually I think you're mistaken here.  "g_checksum_*()" functions are part of GLib, which is different from glibc.  By chance, CentOS 5 uses GLib 2.12, and CentOS 6 uses glibc 2.12... it's very confusing :)

CentOS 6 currently provides GLib 2.28.  I think that would make Manylinux 2010 viable?

[Daniel Alley]

I spun up a CentOS 6 box just to double-check and was able to successfully build createrepo_c

[Jean-Christophe Fillion-Robin]

Excellent. And nice work sorting out the details. That means Manylinux 2010 should work great :)

To view this discussion on the web visit https://groups.google.com/d/msgid/scikit-build/CAEW5TTqL_tqiNU7FrNyCTM9hMsEcVbteq5uNJ%3DtGvg%3DMc18znw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Jean-Christophe Fillion-Robin]

> With respect to sdist problems

I will have a look and report back.

[Jean-Christophe Fillion-Robin]

Hi  Daniel,

Since the source of the python packages are located in the directory "src/python", the package_dir dictionary in setup.py has to be changed from:

package_dir={
    'createrepo_c': 'createrepo_c'
},

to

package_dir={
    'createrepo_c': 'src/python'
},

Similarly, the CMake install rules have to target the same directory.

I updated the PR with a the appropriate fix. See https://github.com/dralley/createrepo_c/pull/1

Let me know if you have any other questions,

Jc

[Daniel Alley]

Jean,

Thank you so much, this has been an enormous help!

I do have one more question: am I doing the right thing in this commit?  It builds and it works, but I'm not much of a C person, so I don't know if I'm missing anything.

https://github.com/rpm-software-management/createrepo_c/pull/103/commits/e880bf07c6ba51018341b58ed91323818735659e

(note: I've rebased my branch so if you make another PR you might run into that)

[Jean-Christophe Fillion-Robin]

Hi Daniel,

You will find here few more changes to ensure that the exectuables are available after installing the wheel: https://github.com/dralley/createrepo_c/pull/2

In a nutshell, the binaries are shipped as data within the "createrepo_c" package.

Here is the content of the wheel:

Archive:  createrepo_c-0.10.0-cp35-cp35m-linux_x86_64.whl
  inflating: Unpack-3104/createrepo_c/__init__.py 
  inflating: Unpack-3104/createrepo_c/_createrepo_c.so 
  inflating: Unpack-3104/createrepo_c/data/bin/createrepo_c 
  inflating: Unpack-3104/createrepo_c/data/bin/mergerepo_c 
  inflating: Unpack-3104/createrepo_c/data/bin/modifyrepo_c 
  inflating: Unpack-3104/createrepo_c/data/bin/sqliterepo_c 
  inflating: Unpack-3104/createrepo_c-0.10.0.dist-info/entry_points.txt 
  inflating: Unpack-3104/createrepo_c-0.10.0.dist-info/top_level.txt 
  inflating: Unpack-3104/createrepo_c-0.10.0.dist-info/WHEEL 
  inflating: Unpack-3104/createrepo_c-0.10.0.dist-info/METADATA 
  inflating: Unpack-3104/createrepo_c-0.10.0.dist-info/RECORD

Then, within the "createrepo_c/__init__.py ", we have four functions allowing to launch each one of the executables:

def _program(name, args):
    return subprocess.call([os.path.join(BIN_DIR, name)] + args)


def createrepo_c():
    raise SystemExit(_program('createrepo_c', sys.argv[1:]))

Then in setup.py, we declared entrypoints to ensure launcher script are created in the right location after installing the wheel:

entry_points={
        'console_scripts': [
            'createrepo_c=createrepo_c:createrepo_c',
            'mergerepo_c=createrepo_c:mergerepo_c',
            'modifyrepo_c=createrepo_c:modifyrepo_c',
            'sqliterepo_c=createrepo_c:sqliterepo_c'
        ]
},

Et voila,

Jc

[Jean-Christophe Fillion-Robin]

Hi Daniel,

One last comment about the layout of the project, while scikit-build is able to create a working wheel, the layout of the source prevents developer from having a usable editable installation.

After running "pip install -e ." or "python setup.py develop" scikit-build makes sure the python module _createrepo_c.so and the binaries are copied into the source tree but they are not usable.

For example, trying to "import createrepo_c" fails because the the package "createrepo_c/__init__.py" does not exist.

It may be sensible to relocate the files from "src/python/*" into "src/python/createrepo_c/*"

Hth

Jc