pyside bindings setuptool'ized build

[Joel B. Mohler]

I have a tentative patch at https://bitbucket.org/jbmohler/scintilla/changeset/494006b615fdca1fb9c3230fc9efbf02fc32e187 which puts a setup.py in the qt subdirectory and implements the setuptools commands enabling egg distribution, uploading to pypi, and install to site-packages.  I have preliminary evidence that it works on both windows and linux with PySide 1.1.2 and Qt 4.8.x.  I don't see any reason it wouldn't work on nearby versions.

The sepbuild.py should work as it always has, but I would view that as deprecated with this patch.  I'm aware that my pyside-base switch doesn't work as sepbuild's does with regards to Qt.  It seems to me that the sepbuild semantics for that are a niche case; a better implementation in my view would be to have independent qt-base and pyside-base parameters which may be set to the same thing if sepbuild's semantics are desired.

I would like to know if this is something that may be accepted.  We almost certainly want an egg and associated tooling for our in-house use of these bindings.  I've also written an __init__ which I think exposes the namespace in a fairly convenient way.

Joel

[John Ehresman]

On 9/21/12 4:29 PM, Joel B. Mohler wrote:
> The sepbuild.py should work as it always has, but I would view that as
> deprecated with this patch. I'm aware that my pyside-base switch
> doesn't work as sepbuild's does with regards to Qt. It seems to me that
> the sepbuild semantics for that are a niche case; a better
> implementation in my view would be to have independent qt-base and
> pyside-base parameters which may be set to the same thing if sepbuild's
> semantics are desired.

Would this script work without setuptools? We use the current
sepbuild.py and would not want to use setuptools. Support for win32 &
OS X and the ability to compile against a non-system wide installation
of both qt and of pyside are also something we'd need.

Thanks,

John

[Joel B. Mohler]

This code relies on setuptools using the setuptools base classes and setup calls.  However I use the same ScintillaEditPy.pro with qmake so the build is essentially the same thing.  I didn't make any effort to share code (given my view toward replacing sepbuild).  Having now seen a complete working setuptools solution for a shiboken project I think I can definitely see a way to share lots of the code perhaps most elegantly by deriving the setuptools stuff from SepBuilder.  By the way, I have a parameter for qmake which I think effectively gives a qt-base parameter since qmake can be queried for paths and such.

Would code sharing with sepbuild.py help this get accepted?

Beside the point, but I'm curious what workflow in the python world requires not using setuptools?  It seems to me a lot of 3rd party stuff assumes setuptools as a basic feature.

Thanks, Joel

[Neil Hodgson]

Joel B. Mohler:

> Would code sharing with sepbuild.py help this get accepted?

The main requirements for acceptance are that it works on all three platforms and that it works well for existing users. Other factors include being reasonably clear and maintainable.

The only current users of Scintilla for PySide I know of are you and Wingware.

Since it may be necessary to support different versions of Qt and PySide on one machine, and possibly variants such as debug/release, it seems worthwhile to include some way of choosing a particular PySide version directory. Or both PySide and Qt directories if the matching Qt installation can't be derived from PySide.

I don't know how PyPi handles variants: it seems that any Scintilla release will have, at least, a block of binaries over Platforms (Win, Linux, OS X) Python versions (2.6, 2.7, 3.2?) * Qt versions with possibly some other variants like 32/64 bit and framework versus non-framework on OS X. It looks like a fair bit of work to build and upload all these.

Neil

[Neil Hodgson]

Joel B. Mohler:

> Would code sharing with sepbuild.py help this get accepted?

After thinking about this some more, I think I am going to have to say that there should only be one way to build Scintilla for PySide included in the Scintilla repository. So, if this setup.py version is to be included, then it should supersede sepbuild.py.

Contributors come and go. While most of the code is wanted enough by someone that they will help with its maintenance, that's not the case for everything. Sometimes this ends up with maintenance falling over to me or going through the messy process of dropping support. Therefore I want to minimize the cost of supporting PySide and the size and complexity of its build processes.

Neil

[Joel B. Mohler]

I understand your concern.  My project is almost certainly using PySide Scintilla and we would provide the necessary maintenance supposing we use it (unresolved questions about project scope will affect the need for an editor).  That said, I can write an out-of-tree python egg maker for myself if it comes to that -- in reality, that's pretty much exactly what I have at this point.

So, here are my objectives (all are my opinion and subject to other opinions) and given this exchange I'm uncertain how to proceed:

1)  sepbuild.py (on windows) assumes PySide in c:\usr or useage --pyside-base in which case Qt includes/libs must be with the pyside-base.  Both of these alternative are unnecessarily subpar.  In fact, it's possible with modest modifications to sepbuild to build Scintilla PySide bindings with nothing but a PySide binary install and an installed Qt opensource package -- no building of PySide is necessary.
2)  The __init__.py, ScintillaConstants.py, ScintillaEditPy.pyd created in the bin subfolder require importing submodules -- I think an "import ScintillaPySide" with a full namespace makes more sense.  I believe my __init__.py makes clear what I have in mind.
3)  I also note that testsepq.py gives an import error if PySide.QtCore is not included before the ScintillaEditPy.pyd -- this is easy to fix when you know it, but it's not nice.  My __init__.py imports PySide.QtCore which addresses this dll location caveat in a predictable manner.  Concretely, when I move the PySide imports below the ScintillaEditPy import, I get the following error:
{{{
Traceback (most recent call last):
  File "C:\mercurial\scintilla\qt\ScintillaEditPy\testsepq.py", line 9, in <module>
    from bin import ScintillaEditPy
ImportError: DLL load failed: The specified module could not be found.
}}}
4)  Peripheral to my primary concerns, but I believe it is feasible and of value to the community to have a pypi package with the ScintillaPySide bindings pre-built for windows.  I believe that windows binaries are of much greater relative importance given the relative difficulty of building in that environment and market share reasons.  I believe that this makes Scintilla usage much more approachable for my team (as an example) although clearly I can fake it myself for us.

Bottom line:  I can achieve my goals with no upstream changes, but it feels hacky.  I believe that there's room for improvement in the PySide binding imports and I'd like to have my code match what is ultimately decided upstream.

Your input on these issues is desired.

Thanks,
Joel

[John Ehresman]

I looked briefly to the proposed setup.py and I think it could be made
to work with setuptools, which is a dependency that I do not want (I am
familiar with setuptools and what it does). I don't think the distutils
only version would build an egg or upload to pypi, but I think it would
build the .pyd/.so files.

On 9/24/12 8:34 AM, Joel B. Mohler wrote:
> So, here are my objectives (all are my opinion and subject to other
> opinions) and given this exchange I'm uncertain how to proceed:
>
> 1) sepbuild.py (on windows) assumes PySide in c:\usr or useage
> --pyside-base in which case Qt includes/libs must be with the
> pyside-base. Both of these alternative are unnecessarily subpar. In
> fact, it's possible with modest modifications to sepbuild to build
> Scintilla PySide bindings with nothing but a PySide binary install and
> an installed Qt opensource package -- no building of PySide is necessary.

We'd be building against our pyside build, which is done via the
--pyside-base option. As long as this still works, allowing builds
against a binary PySide install sounds like a good idea.

> 2) The __init__.py, ScintillaConstants.py, ScintillaEditPy.pyd created
> in the bin subfolder require importing submodules -- I think an "import
> ScintillaPySide" with a full namespace makes more sense. I believe my
> __init__.py makes clear what I have in mind.

Defining a package makes sense to me.

> 4) Peripheral to my primary concerns, but I believe it is feasible and
> of value to the community to have a pypi package with the
> ScintillaPySide bindings pre-built for windows. I believe that windows
> binaries are of much greater relative importance given the relative
> difficulty of building in that environment and market share reasons. I
> believe that this makes Scintilla usage much more approachable for my
> team (as an example) although clearly I can fake it myself for us.

I agree that Windows binaries is something people will want.

Thanks,

John

[Neil Hodgson]

Joel B. Mohler:

> I understand your concern. My project is almost certainly using PySide Scintilla and we would provide the necessary maintenance supposing we use it

Except for OS X or is that also a target for you?

For Linux, both Red Hat based distributions and Debian based distributions should be supported. IIRC it was Fedora which installs qmake as qmake-qt4 and it looks like that will cause the current setup.py to fail (since it tests OPTION_QMAKE before checking with 'which') unless it is told where to look unlike sepbuild.py.

There is a line in the modified sepbuild.py that appears to be Windows-specific as backslashes won't work on Unix:
f.write("LIB_DEST=" + doubleBackSlashes(r'..\..\bin') + "\n")

> 2) The __init__.py, ScintillaConstants.py, ScintillaEditPy.pyd created in the bin subfolder require importing submodules -- I think an "import ScintillaPySide" with a full namespace makes more sense. I believe my __init__.py makes clear what I have in mind.

OK.

> 3) I also note that testsepq.py gives an import error if PySide.QtCore is not included before the ScintillaEditPy.pyd

If it makes it harder to fail then its probably a benefit.

> 4) Peripheral to my primary concerns, but I believe it is feasible and of value to the community to have a pypi package with the ScintillaPySide bindings pre-built for windows. I believe that windows binaries are of much greater relative importance given the relative difficulty of building in that environment and market share reasons.

Are you planning on uploading these yourself?

The politics and support situation around setuptools and distribute has made many wary. Its unfortunate that distutils2 and its integration into core Python has also been delayed past 3.3. I don't follow these things closely but I think setuptools is not compatible with the current plans for distutils2.

Neil

[Joel B. Mohler]

After getting distracted by many things, I had a moment while hunkered down in hurricane Sandy to rewrite this patch to not use setuptools.  This patch is at https://bitbucket.org/jbmohler/scintilla/changeset/dd5e67190dd1d4a6b59c028de27a06c4706c6c80 .  I believe this entirely addresses the concerns with setuptools reliance.

I've uncovered some strangeness in that the official PySide builds appear to be linked in some way to the Qt installation they were built with -- concretely this means that the PySide 1.1.2 windows binary on qt-project.org won't be usable to build with the Qt 4.8.3 downloaded from qt-project.org (I had to use 4.8.2 or build PySide myself).  This seems like an oddity that doesn't really have to do with python bindings for scintilla, but just impacts us in this context.  So, the build instructions are now:
1) install PySide 1.1.2 -- reliant on Qt 4.8.2
2) install Qt SDK 4.8.2 (must be in C:\Qt\4.8.2 on windows)
2) clone https://bitbucket.org/jbmohler/scintilla
3) cd scintilla\hg
4) python setup.py install --qmake=c:\Qt\4.8.2\bin\qmake.exe

As for putting builds on pypi, I would not wish to commit myself to keeping such an item current.  I was more assuming that it would be something that is amenable to some automation which is attached to the windows binary production of scintilla under the assumption that such mechanism exists for someone.  My thinking is that building this is really not very pleasant for a pure python developer on windows using scintilla (although that's a pretty small piece of any applicable venn diagram to be sure!!)

Feedback is appreciated ...

Joel

[John Ehresman]

On 10/31/12 3:49 PM, Joel B. Mohler wrote:
> After getting distracted by many things, I had a moment while hunkered
> down in hurricane Sandy to rewrite this patch to not use setuptools.
> This patch is at
> https://bitbucket.org/jbmohler/scintilla/changeset/dd5e67190dd1d4a6b59c028de27a06c4706c6c80
> . I believe this entirely addresses the concerns with setuptools reliance.

I will look at in the next few days.

> I've uncovered some strangeness in that the official PySide builds
> appear to be linked in some way to the Qt installation they were built
> with

I recently discovered this as well; it seems to be that shiboken picks
up some hardcoded paths when it is built.

> As for putting builds on pypi, I would not wish to commit myself to
> keeping such an item current.

I can't commit to supporting this either. Windows developers will want
binary distributions, but I also think it will be no small amount of
work to keep them current and provide support for the developers using
the binaries. I think a setup.py like the one proposed might help
someone package and upload the binaries, but there's probably more work
to be done.

Thanks,

John

[Noli Sicad]

Hi,

I decided to upgrade to Qt 4.8.0 and PySide 1.1.1 Again, I am using
Mac OS X 10.8.2.

Got an error as well (below).

Thanks, Noli

##################

Nolis-MacBook-Pro:qt nsicad$ qmake --version
QMake version 2.01a
Using Qt version 4.8.0 in /Library/Frameworks


Nolis-MacBook-Pro:qt nsicad$ python setup.py build
running build
shiboken --generator-set=shiboken global.h --avoid-protected-hack
--enable-pyside-extensions
--include-paths=/usr/include:/usr/include/QtCore:/usr/include/QtGui:/Users/nsicad/Documents/Python_Nolis/scintilla/qt/ScintillaEditPy:/Users/nsicad/Documents/Python_Nolis/scintilla/qt/ScintillaEdit:/Users/nsicad/Documents/Python_Nolis/scintilla/qt/ScintillaEditBase:/Users/nsicad/Documents/Python_Nolis/scintilla/qt/../include:/Library/Python/2.7/site-packages/PySide/include/shiboken:/Library/Python/2.7/site-packages/PySide/include/PySide:/Library/Python/2.7/site-packages/PySide/include/PySide/QtCore:/Library/Python/2.7/site-packages/PySide/include/PySide/QtGui
--typesystem-paths=/Library/Python/2.7/site-packages/PySide/typesystems
--output-directory=. typesystem_ScintillaEdit.xml
Fatal error: line=3, column=60, message=Failed to parse: 'typesystem_core.xml'

Cannot parse file: typesystem_ScintillaEdit.xml('Failed in shiboken', 1)




Nolis-MacBook-Pro:qt nsicad$ python setup.py install
running install
running build
shiboken --generator-set=shiboken global.h --avoid-protected-hack
--enable-pyside-extensions
--include-paths=/usr/include:/usr/include/QtCore:/usr/include/QtGui:/Users/nsicad/Documents/Python_Nolis/scintilla/qt/ScintillaEditPy:/Users/nsicad/Documents/Python_Nolis/scintilla/qt/ScintillaEdit:/Users/nsicad/Documents/Python_Nolis/scintilla/qt/ScintillaEditBase:/Users/nsicad/Documents/Python_Nolis/scintilla/qt/../include:/Library/Python/2.7/site-packages/PySide/include/shiboken:/Library/Python/2.7/site-packages/PySide/include/PySide:/Library/Python/2.7/site-packages/PySide/include/PySide/QtCore:/Library/Python/2.7/site-packages/PySide/include/PySide/QtGui
--typesystem-paths=/Library/Python/2.7/site-packages/PySide/typesystems
--output-directory=. typesystem_ScintillaEdit.xml
Fatal error: line=3, column=60, message=Failed to parse: 'typesystem_core.xml'

Cannot parse file: typesystem_ScintillaEdit.xml('Failed in shiboken', 1)
Nolis-MacBook-Pro:qt nsicad$

[Neil Hodgson]

Joel B. Mohler:

> 2) clone https://bitbucket.org/jbmohler/scintilla
> 3) cd scintilla\hg

Should be
cd scintilla\qt

> 4) python setup.py install --qmake=c:\Qt\4.8.2\bin\qmake.exe

OK, this works on Windows.

For both Linux and OS X, there were two long lines in qt/ScintillaEditPy/ScintillaEditPy.pro which have been wrapped but should not be. Patch attached.