Bug#1051774: python3-pysnmp4: Asyncio backend is incompatible with default Python 3.11

[Adam Cecile]

Package: python3-pysnmp4
Version: 4.4.12-2
Severity: important

Hello,

Current version shipped with Debian 12 is partially broken, asyncio backend is
using deprecated feature that have been removed from Python 3.11:

Traceback (most recent call last):
File "/home/acecile/dev/c/ltms/monitoring/check-ntcip-road-
sign/check_ntcip_road_sign.py", line 16, in <module>
from pysnmp.hlapi.asyncio import SnmpEngine, getCmd, CommunityData,
UdpTransportTarget, ContextData, ObjectType, ObjectIdentity
File "/usr/lib/python3/dist-packages/pysnmp/hlapi/asyncio/__init__.py", line
12, in <module>
from pysnmp.hlapi.asyncio.transport import *
File "/usr/lib/python3/dist-packages/pysnmp/hlapi/asyncio/transport.py", line
9, in <module>
from pysnmp.carrier.asyncio.dgram import udp, udp6
File "/usr/lib/python3/dist-packages/pysnmp/carrier/asyncio/dgram/udp.py",
line 35, in <module>
from pysnmp.carrier.asyncio.dgram.base import DgramAsyncioProtocol
File "/usr/lib/python3/dist-packages/pysnmp/carrier/asyncio/dgram/base.py",
line 36, in <module>
from pysnmp.carrier.asyncio.base import AbstractAsyncioTransport
File "/usr/lib/python3/dist-packages/pysnmp/carrier/asyncio/base.py", line
33, in <module>
from pysnmp.carrier.asyncio.dispatch import AsyncioDispatcher
File "/usr/lib/python3/dist-packages/pysnmp/carrier/asyncio/dispatch.py",
line 46, in <module>
class AsyncioDispatcher(AbstractTransportDispatcher):
File "/usr/lib/python3/dist-packages/pysnmp/carrier/asyncio/dispatch.py",
line 57, in AsyncioDispatcher
@asyncio.coroutine
^^^^^^^^^^^^^^^^^

After looking at GitHub, here is what I figured out:

* Upstream maintainer of pySNMP4 has passed away so no more update are being
done (https://github.com/etingof/pysnmp/issues/429)
* A new upstream seems to have taken over the project
(https://github.com/lextudio/pysnmp)
* It is probably possible to backport a couple of asyncio fix to get the
package working with Python 3.11, I may be able to help but I'm not sure if
this is the way to go
(https://github.com/lextudio/pysnmp/commits/main/pysnmp/carrier/asyncio/dispatch.py)

In my opinion the bug is serious enough to require a fix for Debian 12, but it
is not my call. I'll try to backport asyncio fixes from new upstream into
stable package to see if it helps.

Best regards, Adam.


-- System Information:
Debian Release: 12.1
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.1.0-11-amd64 (SMP w/12 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages python3-pysnmp4 depends on:
ii python3 3.11.2-1+b1
ii python3-pyasn1 0.4.8-3
ii python3-pycryptodome 3.11.0+dfsg1-4
ii python3-pysmi 0.3.2-3

python3-pysnmp4 recommends no packages.

python3-pysnmp4 suggests no packages.

-- no debconf information

[Adam Cecile]

Hello,

So it turns out there was two issues here:

* New "lextudio" upstream patch broke asyncio support by converting regular function returning future into awaitable function returning Future (double await needed).

I fixed the issue and send a PR upstream: https://github.com/lextudio/pysnmp/pull/24

Bug was already reported but not taken in account: https://github.com/lextudio/pysnmp/issues/19

* Upstream "lextudio" patches to fix asyncio backend (including my own PR from today) had to be merged into debian package.

I created an upstream based branch here to see what patches have been cherry-picked: https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commits/4.4.12+cherry-pick-asyncio-lextudio-fixes/

And created a Debian 4.4.12-3 release so I can build and test the package: https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commit/a5f17d27c7813dbdb64cdf674d1855a77c3eb0f0

I'll now try to reach Debian Python team to see if we should proceed further.

Regards, Adam.

[Thomas Goirand]

On 9/12/23 18:16, Adam Cecile wrote:
> Hello,
>
> No hurry, I think we might want to wait for upstream to respond to my PR
> regarding double awaitable fix.
> It is indeed lextudio upstream that took over the PySNMP package and all
> patches are coming from us (except mine ofc).
>
> Regards, Adam.

Because it messes up the order in which people normally read text.
Why is top-posting such a bad thing?
Top-posting.
What is the most annoying thing in e-mail?

Thanks! :)

I tried applying your patch at
https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commit/88d40f1225de8f7b42413b56206b41a6155fcf09

Unfortunately, it doesn't apply on top of 4.4.12-2, which is the current
version of the package (in Bookworm, Unstable and Testing).

Would you be able to rebase your patch on top of 4.4.12-2? Then I'll do
the work to get this into Bookworm (and Unstable/Testing).

Cheers,

Thomas Goirand (zigo)

[Adam Cecile]

On 9/13/23 12:55, Thomas Goirand wrote:

On 9/12/23 18:16, Adam Cecile wrote:

Hello,

No hurry, I think we might want to wait for upstream to respond to my PR regarding double awaitable fix.
It is indeed lextudio upstream that took over the PySNMP package and all patches are coming from us (except mine ofc).

Regards, Adam.

Because it messes up the order in which people normally read text.
Why is top-posting such a bad thing?
Top-posting.
What is the most annoying thing in e-mail?

Hello, you started first !

Thanks! :)

I tried applying your patch at https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commit/88d40f1225de8f7b42413b56206b41a6155fcf09

Unfortunately, it doesn't apply on top of 4.4.12-2, which is the current version of the package (in Bookworm, Unstable and Testing).

Would you be able to rebase your patch on top of 4.4.12-2? Then I'll do the work to get this into Bookworm (and Unstable/Testing).

Cheers,

Thomas Goirand (zigo)

Yes that's expected. This commit is only to fix double awaitable "new" upstream bug. It depends on a large amount of backported commits to fix asyncio / Python 3.11 support.

You can see here a branch created from upstream 4.4.12 tag with asyncio patches cherry-pick from new upstream master:

https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commits/4.4.12+cherry-pick-asyncio-lextudio-fixes/

It has then been squashed into a single debian/patch:

https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commit/a5f17d27c7813dbdb64cdf674d1855a77c3eb0f0

I made my own forked repository because I'm unsure how we should proceed, but I can easily push the debian/4.4.12-3 tag to the regular Python module repository on Salsa.

Adam.

[Thomas Goirand]

On 9/13/23 13:43, Adam Cecile wrote:
> On 9/13/23 12:55, Thomas Goirand wrote:
>> On 9/12/23 18:16, Adam Cecile wrote:
>>> Hello,
>>>
>>> No hurry, I think we might want to wait for upstream to respond to my
>>> PR regarding double awaitable fix.
>>> It is indeed lextudio upstream that took over the PySNMP package and
>>> all patches are coming from us (except mine ofc).
>>>
>>> Regards, Adam.
>>
>> Because it messes up the order in which people normally read text.
>> Why is top-posting such a bad thing?
>> Top-posting.
>> What is the most annoying thing in e-mail?
> Hello, you started first !

LOL ! :)

Well, I was on my phone, sorry for that ... :P

>> Thanks! :)
>>
>> I tried applying your patch at
>> https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commit/88d40f1225de8f7b42413b56206b41a6155fcf09
>>
>> Unfortunately, it doesn't apply on top of 4.4.12-2, which is the
>> current version of the package (in Bookworm, Unstable and Testing).
>>
>> Would you be able to rebase your patch on top of 4.4.12-2? Then I'll
>> do the work to get this into Bookworm (and Unstable/Testing).
>>
>> Cheers,
>>
>> Thomas Goirand (zigo)
>
> Yes that's expected.

Well, how can I then apply it to the version in Bookworm?

> This commit is only to fix double awaitable "new"
> upstream bug. It depends on a large amount of backported commits to fix
> asyncio / Python 3.11 support.

Could you backport it to 4.4.12-2 as in Bookworm and Unstable?

As I wrote already, I already packaged python-pysnmp-lextudio, which is
currently in the NEW queue. I will be happy to apply your patch in
there, but IMO, we should treat pysnmp-lextudio as a different source
and binary package (my binary conflicts with python3-pysnmp4), because
the dependency chain is very different.

> You can see here a branch created from upstream 4.4.12 tag with asyncio
> patches cherry-pick from new upstream master:
>
> https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commits/4.4.12+cherry-pick-asyncio-lextudio-fixes/
>
> It has then been squashed into a single debian/patch:
>
> https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commit/a5f17d27c7813dbdb64cdf674d1855a77c3eb0f0

Ah, super cool! It's too late for today (have to go back home), so I'll
work on this tomorrow. Thanks a lot for your contrib.

BTW, we've been using your MegaCli repo (we mirror it), and I also would
like to thank you for this. :)

> I made my own forked repository because I'm unsure how we should
> proceed, but I can easily push the debian/4.4.12-3 tag to the regular
> Python module repository on Salsa.

4.4.12-3 will be for Unstable. For Stable, it's going to be something
like 4.4.12-2+deb12u1, as per the normal process, and it will have to be
(pre-)approved by the Debian Stable release team by filling a bug
against release.debian.org. No worries, I do understand that Debian
procedures are not easy to understand, though I'm happy to explain if
you need.

Cheers,

Thomas Goirand (zigo)

[Adam Cécile]

Hello,

Soory for the delay, I don't get the question, bookworm version is the
same as unstable at the moment so my debian/4.4.12-3 branch also works:

https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commits/debian/4.4.12-3

If you want only the full patch fixing asyncio, you can find it as
debian/patch:

https://salsa.debian.org/acecile-guest/python-pysnmp4/-/blob/debian/4.4.12-3/debian/patches/0003-Merge-lextudio-upstream-fork-patch-related-to-asynci.patch

>
>> This commit is only to fix double awaitable "new" upstream bug. It
>> depends on a large amount of backported commits to fix asyncio /
>> Python 3.11 support.
>
> Could you backport it to 4.4.12-2 as in Bookworm and Unstable?
>
> As I wrote already, I already packaged python-pysnmp-lextudio, which
> is currently in the NEW queue. I will be happy to apply your patch in
> there, but IMO, we should treat pysnmp-lextudio as a different source
> and binary package (my binary conflicts with python3-pysnmp4), because
> the dependency chain is very different.

Yes it's already done, see above.

>
>> You can see here a branch created from upstream 4.4.12 tag with
>> asyncio patches cherry-pick from new upstream master:
>>
>> https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commits/4.4.12+cherry-pick-asyncio-lextudio-fixes/
>>
>>
>> It has then been squashed into a single debian/patch:
>>
>> https://salsa.debian.org/acecile-guest/python-pysnmp4/-/commit/a5f17d27c7813dbdb64cdf674d1855a77c3eb0f0
>>
>
> Ah, super cool! It's too late for today (have to go back home), so
> I'll work on this tomorrow. Thanks a lot for your contrib.

So, all good?

>
> BTW, we've been using your MegaCli repo (we mirror it), and I also
> would like to thank you for this. :)

Thanks! Sadly I miss time to take care of it, but no matter how old and
badly written was the Python code, it still works flawlessly :-) Cheers
to LSI/Broadcom for not breaking tools and output format btw.