CY2024 - Python 3.12?

[Michael Rochefort]

As of now the CY2024 draft specification designates 3.11.x as the target Python version. Is there any appetite for bumping this to Python 3.12.x?

https://docs.python.org/dev/whatsnew/3.12.html

Some developer and performance benefits I can see from this:

* Support for Linux perf profiling of Python code (Linux only)[1]
* Inlined comprehension for up to 2x performance improvement[2]
* Ability to profile and trace all running Python threads, not just the calling thread hierarchy[3]

There are a lot of minor changes and additions in this release as well, but their impact will be localized to developer usage. However, it would be good to have a comprehensive review around the impact of the standard library removals[4] and cAPI changes[5].

Besides any potential incompatibilities or significant lift required to target Python 3.12, there are two primary issues I see with bumping: availability[6] and ecosystem support.

The PSF is targeting to have release candidate versions available for testing July 31st, and the first beta was released yesterday (no new features from now on). The GA, though, won't be available until October 2nd, which may be too late for the platform to accept. And it's uncertain if any of the changes in 3.12 will require extensive compatibility testing and validation for a number of ecosystem packages that vendors will not have enough time to complete.

Thoughts?

[1] https://docs.python.org/dev/howto/perf_profiling.html
[2] https://peps.python.org/pep-0709/
[3] https://docs.python.org/dev/whatsnew/3.12.html#threading
[4] https://docs.python.org/dev/whatsnew/3.12.html#removed
[5] https://docs.python.org/dev/whatsnew/3.12.html#c-api-changes
[6] https://peps.python.org/pep-0693/

Cheers,
Mike

[Neal Gompa]

On Tue, May 23, 2023 at 2:07 PM 'Michael Rochefort' via
vfx-platform-discuss <vfx-platfo...@googlegroups.com> wrote:
>
> As of now the CY2024 draft specification designates 3.11.x as the target Python version. Is there any appetite for bumping this to Python 3.12.x?
>
> https://docs.python.org/dev/whatsnew/3.12.html
>
> Some developer and performance benefits I can see from this:
>
> * Support for Linux perf profiling of Python code (Linux only)[1]
> * Inlined comprehension for up to 2x performance improvement[2]
> * Ability to profile and trace all running Python threads, not just the calling thread hierarchy[3]
>
> There are a lot of minor changes and additions in this release as well, but their impact will be localized to developer usage. However, it would be good to have a comprehensive review around the impact of the standard library removals[4] and cAPI changes[5].
>
> Besides any potential incompatibilities or significant lift required to target Python 3.12, there are two primary issues I see with bumping: availability[6] and ecosystem support.
>
> The PSF is targeting to have release candidate versions available for testing July 31st, and the first beta was released yesterday (no new features from now on). The GA, though, won't be available until October 2nd, which may be too late for the platform to accept. And it's uncertain if any of the changes in 3.12 will require extensive compatibility testing and validation for a number of ecosystem packages that vendors will not have enough time to complete.
>
> Thoughts?
>

The perf stuff doesn't do you much good unless you're using a platform
that is compiled with frame pointers enabled. As of today, the only
distribution that does that is Fedora with Fedora Linux 38 and newer:
https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer

Currently, it is disabled in ELN (which is the protoform used as an
input for creating CentOS Stream 10):
https://src.fedoraproject.org/rpms/redhat-rpm-config/c/9e10bb665ec2d3cc4b1ea613a7fe8e99e467e64e

I don't know if they will enable it, that may depend on requests for
the capability. But this capability missing affects the ability to
profile and trace Python threads too. While it's possible to compile
Python itself with frame pointers (and I believe upstream will start
recommending that), anything that calls outside of Python would be
missing that information. Given how much Python is used as "glue"
between C/C++ libraries and applications, that can be a problem.

For more info about the utility and usability of this stuff, this blog
post is a decent starting point:
https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-dwarf-my-verdict/

So when you leave out that stuff, you're not left with much to make it
an attractive upgrade.



--
真実はいつも一つ！/ Always, there's only one truth!

[Deke Kincaid]

Here is the release schedule for Python 3.12

https://peps.python.org/pep-0693/

This comes up every single year.  Commercial software vendors and open source projects need time to get this into their packages they release in 2024.  If the release does not come out till October, that leaves very little time to properly integrate and test your software from a major dependency change.  Many vendors already have alpha/beta builds going by the end of the year.  Your alphas should have all dependency changes mostly done and by beta there should be no dependencies changing.  Something as fundamental as a new Python version which doesn't even have a few patch releases under its belt is a big risk.  Especially when half your users often say "just make a version that crashes less and is more stable".  Things like this do the opposite.

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/18849cb1626.c5952f7b247845.5155728950970422478%40michaelrochefort.com.

[Michael Rochefort]

Thanks for the info on the perf and frame pointers support, Neal!

On Tuesday, May 23, 2023 at 5:00:37 PM UTC-4 dekek...@gmail.com wrote:

Commercial software vendors and open source projects need time to get this into their packages they release in 2024.  If the release does not come out till October, that leaves very little time to properly integrate and test your software from a major dependency change.  Many vendors already have alpha/beta builds going by the end of the year.

This is along the lines of what I was saying in my original post, and you're right that it does come up every year and Nick pointed out the last time that the Sept 1st deadline has become much stricter for selections (excluding the industry projects that are granted exceptions). Playing devils advocate from one perspective, Python 3.11 will be in a security fixes-only state by the end of this year, so lingering bugs and issues (if any that impact software providers) will not be fixed at all for CY2024 compatible tooling.

> 3.11 will receive bugfix updates approximately every 2 months for approximately 18 months. Some time after the release of 3.12.0 final, the ninth and final 3.11 bugfix update will be released. After that, it is expected that security updates (source only) will be released until 5 years after the release of 3.11.0 final, so until approximately October 2027.

https://peps.python.org/pep-0664/

However, the flip side can also be considered and ties into your other point.

 

Something as fundamental as a new Python version which doesn't even have a few patch releases under its belt is a big risk.  Especially when half your users often say "just make a version that crashes less and is more stable".

 

Having a "mature" product to build off of where all bugs/issues are known and workarounds can be implemented and shared is also a valid mindset. I'm neither for nor against either option, just posing the question.

Going back to the other side, in terms of vendor preparation if you push the third-party package ecosystem out of the picture realistically vendors could start validating 3.12 (if the schedule is kept) on Aug 1 with the first release candidate. That would provide five months of testing and validation to ensure problems are worked out (three with actual 3.12 GA releases) before Jan 1st, and I don't recall any vendors off the top of my head who release early Q1. However, are vendors comfortable with releasing their applications with an x.0, x.1, or x.2 release? I can't answer that question.

Ultimately, if the Reference Platform community is okay with rubber stamping the Sep 1st deadline and requiring a component's GA by then, we should probably codify or set expectations around Python version (I don't recall if we've done this in the past). Based on release schedules, Python is fairly consistent with a GA in the month of October so the selection for any given CY specification would be N-1 of the Python version available by the end of the ratifying year, unless of course there's a reason to continue using a release for multiple CYs.

Cheers,

Mike

[Larry Gritz]

What we need to appreciate is that when we say that a package like OpenEXR needs to have their release by October 202x to be included in VFX Platform 202x+1, we are already building in consideration for the fact that this package is being developed primarily by engineers at the big vfx studios, so some of them have been testing it, and probably using it in real production software on real shows, long before that October release. And also that if anything turns out to be badly broken, their own companies and shows are also on the hook and it needs to be fixed right f'ing now.

That is an entirely different situation from a "3rd party" dependency, which was developed outside of production studio needs, almost certainly has not been tested in production studios and their software ecosystem before it's released, and for which if anything is broken, they may or may not consider us a priority. That needs a LOT more lead time across our ecosystem to get to the point where we can merge it in as a key base-level technology on which many other components are built.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAOOm49RZDQphgs-srgR3JhxJ6xHBBmD%2BedHaJcecYLD%3DMFoMag%40mail.gmail.com.

--

Larry Gritz

l...@larrygritz.com