Bug#1008849: Needs tighter dependency on python3

[Yuri D'Elia]

Package: libshiboken2-py3-5.15
Version: 5.15.2-2+b2
Severity: important

shiboken2 depends on a sepecific major+minor version of python.

/usr/lib/x86_64-linux-gnu/cmake/Shiboken2-5.15.2/shiboken_helpers.cmake
checks for this.

As a result, attempting to build any package cmake package using
shiboken2 currently fails:

-- Found PythonInterp: /usr/bin/python3.10 (found suitable version "3.10.4", minimum required is "3")
CMake Error at /usr/lib/x86_64-linux-gnu/cmake/Shiboken2-5.15.2/shiboken_helpers.cmake:468 (message):
The detected Python minor version is not compatible with the Python minor
version which was used when Shiboken was built. Consider building shiboken
with FORCE_LIMITED_API set to '1', so that only the Python major version
matters.

Built with: '3.9' Detected: '3.10'

due to python3 now defaulting to python3.10.

One can set PYTHON_EXECUTABLE to the proper version, but that implies
that libshiboken2-py3-5.15 should depend on a specific python3 version,
not just on any minor version.

This also requires that shiboken2 should be currently rebuilt.

Either the python3 dependency should be tightened, or FORCE_LIMITED_API
should be used.

[Nicholas D Steeves]

Hi Yuri,

Yuri D'Elia <wav...@thregr.org> writes:
[snip]

> Built with: '3.9' Detected: '3.10'
>
> due to python3 now defaulting to python3.10.
>

Thank you for reporting this bug :-)

[snip]

>
> This also requires that shiboken2 should be currently rebuilt.
>

Agreed.

> Either the python3 dependency should be tightened, or FORCE_LIMITED_API
> should be used.

These are not the only available options. Are you aware of binNMUs?
https://wiki.debian.org/binNMU

Regards,
Nicholas

[Yuri D'Elia]

On Tue, Apr 12 2022, Nicholas D Steeves wrote:
>> This also requires that shiboken2 should be currently rebuilt.
>>
>
> Agreed.
>
>> Either the python3 dependency should be tightened, or FORCE_LIMITED_API
>> should be used.
>
> These are not the only available options. Are you aware of binNMUs?
> https://wiki.debian.org/binNMU

I'm using it for development, and not being able to work if the minor
version changes is a bigger problem than it sounds: I cannot rebuild
third-party software which is using shiboken/pyside unless I completely
rebuild pyside in parallel.

I remember I already had this issue with the 3.9 transition.

IMHO if shiboken2 is tied to the minor version, it should depend on the
minor version as well so that the whole pyside/shiboken2 so that
transitions and downstream dependencies will also be handled as part of
the transitions?

For example, the current situation is probably breaking packages which
are currently build with shiboken.

This makes shiboken2 completely useless if you have more than a one
minor version of python installed (for example during transition
periods). shiboken2 will work only for _one_ minor version.

I actually don't use shiboken2 _directly_ myself, but is there any real
drawback to build with FORCE_LIMITED_API ?

[Nicholas D Steeves]

Hi Yuri and Kurt,

Kurt, would you please read the following discussion, and comment if
it's possible to generate a tighter Python dependency at build time?

Yuri D'Elia <wav...@thregr.org> writes:

> On Tue, Apr 12 2022, Nicholas D Steeves wrote:
>>> This also requires that shiboken2 should be currently rebuilt.
>>>
>>
>> Agreed.
>>
>>> Either the python3 dependency should be tightened, or FORCE_LIMITED_API
>>> should be used.
>>
>> These are not the only available options. Are you aware of binNMUs?
>> https://wiki.debian.org/binNMU
>
> I'm using it for development, and not being able to work if the minor
> version changes is a bigger problem than it sounds: I cannot rebuild
> third-party software which is using shiboken/pyside unless I completely
> rebuild pyside in parallel.
>

I understand, and agree that the issue is real, and that a rebuild is
required. A binNMU is the most expedient solution ;-) Please read
"What are binNMUs?" at the link provided above...

> I remember I already had this issue with the 3.9 transition.
>

Yes, this is totally normal during transitions, and this is why
transitions are needed. You can also see proof of past binNMUs here:

https://snapshot.debian.org/binary/libshiboken2-py3-5.15/

Those "+b" versions are binNMUs.

> IMHO if shiboken2 is tied to the minor version, it should depend on the
> minor version as well so that the whole pyside/shiboken2 so that
> transitions and downstream dependencies will also be handled as part of
> the transitions?
>
> For example, the current situation is probably breaking packages which
> are currently build with shiboken.
>

Kurt, this is the point I'm wondering about, because it would be better
if the transition tracker would alert us of outstanding issues rather
than waiting for someone to report breakage. If this isn't feasible,
could you document that fact in the source package?

> This makes shiboken2 completely useless if you have more than a one
> minor version of python installed (for example during transition
> periods). shiboken2 will work only for _one_ minor version.
>

This might be by design. Kurt, do you know? There's also the question
of if pyside2/shiboken2 is even py3.10-ready (I haven't tested yet).

> I actually don't use shiboken2 _directly_ myself, but is there any real
> drawback to build with FORCE_LIMITED_API ?

As a general principle, I worry that this would either reduce
functionality and/or debugging, or introduce regression potential, so
this is not a change I'm willing to make as a team member and not
as one of the primary maintainers/uploaders of pyside2. I can however
test py3.10 compatibility and request a binNMU.

Best,
Nicholas

[Yuri D'Elia]

On Wed, Apr 13 2022, Nicholas D Steeves wrote:
> I understand, and agree that the issue is real, and that a rebuild is
> required. A binNMU is the most expedient solution ;-) Please read
> "What are binNMUs?" at the link provided above...

Yes, but I wouldn't call this "expedient", since ...

> Kurt, this is the point I'm wondering about, because it would be better
> if the transition tracker would alert us of outstanding issues rather
> than waiting for someone to report breakage. If this isn't feasible,
> could you document that fact in the source package?

... we're waiting for somebody to report the unavoidable breakage, since
shiboken2 enforces this through a minor version check (we're not hoping
it will work - it will just refuse to work).

That would be a "grave" bug report from that perspective, not too
dissimilar to any ABI breakage rendering downstream users _and_ packages
unusable until fixed.

> This might be by design. Kurt, do you know? There's also the question
> of if pyside2/shiboken2 is even py3.10-ready (I haven't tested yet).

Answering this for an hypothetical 3.11 transition, the answer would
similarly be "likely doesn't matter - it's worth attempting a build and
hope for the best, because the current version is broken".

> As a general principle, I worry that this would either reduce
> functionality and/or debugging, or introduce regression potential, so
> this is not a change I'm willing to make as a team member and not
> as one of the primary maintainers/uploaders of pyside2.

I understand this. And the documentation around this define is lacking
as well. Looking at the sources, it does seem to reduce the number of
exported methods, so it is fair to say we might have users that expect
the full API to be available and would break if used...

[Nicholas D Steeves]

Control: retitle -1 shiboken2 - shiboken_helpers.cmake breaks with every Python transition

Disclaimer: I didn't check to see if upstream 5.15.3 fixed this issue.

Yuri D'Elia <wav...@thregr.org> writes:

> On Wed, Apr 13 2022, Nicholas D Steeves wrote:
>> I understand, and agree that the issue is real, and that a rebuild is
>> required. A binNMU is the most expedient solution ;-) Please read
>> "What are binNMUs?" at the link provided above...
>
> Yes, but I wouldn't call this "expedient", since ...
>

Expedient means fast. [1. Optionally check what happens locally]
2. Schedule a binNMU.

Unfortunately it won't work in this case. The pyside2 build currently
in unstable correctly iterated over supported Python versions, and both
libshiboken2-py3-5.15 and libshiboken2-dev contain both Python 3.9 and
3.10 variants...but that's not good enough, because
shiboken_helpers.cmake doesn't appear to support multiple Python
versions, and isn't choosing the right (ie: Debian's python3-default)
version.

Yuri, I see what you mean now, it seems the fastest way to resolve this
would be to statically declare a dependency on python3.10-dev everywhere
in the package, but by doing this the Debian pyside2 package would lose
early detection of FTBFS and failing unit tests with future Python
versions, which means upstream might not receive early enough
notification, which means pyside2 would risk being cut from the next
Debian release if a Python transition happens right before the freeze.
The backported py3.10 support patches already in the pyside2 package are
evidence that the existing approach has value.

[snip]

> Answering this for an hypothetical 3.11 transition, the answer would
> similarly be "likely doesn't matter - it's worth attempting a build and
> hope for the best, because the current version is broken".

And with the above context, your pragmatic point makes sense in a
"perfect is the enemy of the good" sense :-)

Unfortunately your proposed resolution won't solve what now appears to
be the root of the problem: shiboken_helpers.cmake doesn't appear to
support multiple installed Python versions, and will necessarily break
with every Python transition. It seems to me that that cmake file
should be generated during pyside2's build to enable runtime detection
of the support for all Python versions that were compiled into the
pyside2 libraries. Alternatively, as a less desirable option, that
cmake file could be modified in override_dh_auto_install (or somewhere
more appropriate) to exclusively support the current python3-default
version. In both cases, I'm assuming that the compiled py39 and py310
libs are functional.

>> As a general principle, I worry that this would either reduce
>> functionality and/or debugging, or introduce regression potential, so
>> this is not a change I'm willing to make as a team member and not
>> as one of the primary maintainers/uploaders of pyside2.
>
> I understand this. And the documentation around this define is lacking
> as well. Looking at the sources, it does seem to reduce the number of
> exported methods, so it is fair to say we might have users that expect
> the full API to be available and would break if used...

Thank you for your help investigating this option!

Unfortunately I'm out of time for the near future, but if you'd like I
can upload an untested 5.15.3 package to experimental for you to test.
To be honest, I won't have time to make the cmake fix for what isn't
even one of my packages. Sorry.

Sincerely,
Nicholas

[Yuri D'Elia]

On Thu, Apr 21 2022, Nicholas D Steeves wrote:
> Unfortunately I'm out of time for the near future, but if you'd like I
> can upload an untested 5.15.3 package to experimental for you to test.

I can definitely help testing this.

> To be honest, I won't have time to make the cmake fix for what isn't
> even one of my packages. Sorry.

Maybe this is something that upstream would be willing to investigate?

(yeah, I know that all the rage now is to have pyenvs within dockers so
you have 20 copies of everything .. so I'm not holding my breath ;))

[Yuri D'Elia]

On Thu, Apr 21 2022, Yuri D'Elia wrote:
> On Thu, Apr 21 2022, Nicholas D Steeves wrote:
>> Unfortunately I'm out of time for the near future, but if you'd like I
>> can upload an untested 5.15.3 package to experimental for you to test.
>
> I can definitely help testing this.

Ping! ;)