CY2023 Draft Published

[Nick Cannon]

The Draft CY2023 VFX Reference Platform has been published at https://vfxplatform.com.

Major changes are focused on compiler upgrades, and raising glibc together with moving to the new libstdc++ ABI on Linux. This will help the community move away from CentOS Linux before its end of life in 2024.

At our annual Birds of a Feather event this summer the VFX Linux Task Force will present their work along with a recommendation for which Linux distribution to align on over the next couple of years.

The upgrades to OneTBB and Qt 6 will require significant work for some so we are flagging in advance that these are planned for CY2024.

We have already heard requests for a move to C++20. The reason for not yet making this change is that C++20 support in Apple's clang is not yet complete so we are doing further research on this topic and would welcome further input on whether this is a major issue or not.

Please send any feedback on this Draft Platform either to feed...@vfxplatform.com or reply to this thread.

We are looking to lock down the Final version at the start of September after discussing it at the virtual Birds of a Feather event which we are planning for the week of August 1st, the week before the in-person SIGGRAPH conference.

Nick

[Neal Gompa]

This might be a stupid question, but why not upgrade to glibc 2.34
(EL9) rather than glibc 2.28 (EL8)? You're already requiring GCC 11,
which is what EL9 uses. And at least from the EL ecosystem
perspective, the Fedora KDE SIG has shipped Qt 6 to EPEL for
CentOS/RHEL 9 already[1]. It also lengthens the total lifespan for the
platform base (2022-2032 vs 2018-2028). RHEL 9 itself will GA any day
now, and CentOS Stream 9 has been available for the past year.

(Yes, I know that not everyone uses the system provided Qt and other
components, but I think it's helpful if things like this are available
for regular folks like myself trying to build stuff around the VFX
reference platform compatible components.)

I can say as an ISV and an open source developer that working with the
EL9 platform has been drastically easier than the EL8 platform, and I
feel like both system administrators and software developers alike
would prefer the EL9 platform over the EL8 one.

[1]: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-0a30961ad5




--
真実はいつも一つ！/ Always, there's only one truth!

[Larry Gritz]

On May 9, 2022, at 8:24 AM, Nick Cannon <nick....@gmail.com> wrote:

We have already heard requests for a move to C++20. The reason for not yet making this change is that C++20 support in Apple's clang is not yet complete so we are doing further research on this topic and would welcome further input on whether this is a major issue or not.

The way I see it is like this:

Developers not on MacOS (e.g., internal development at a Linux-based studio) don't care.

People who do use MacOS, or who provide libraries that will be consumed by others on that platform, probably do CI or regular builds on all the platforms they support, and thus will immediately catch any accidental use in their products of the unsupported C++20 features. So maybe there are a few C++20 features that they quickly learn to avoid for portability? (And also, we have most of a year for Apple to update their developer tools, which will almost certainly fill in many of the gaps.)

I'm much more concerned with the ABI change, I think that's going to be incredibly disruptive. Though also inevitable. I'm really not sure how to navigate that, since it will mean link-incompatibility with all the standard packages installed with the OS. I can only assume that this means everybody has to make a hard break away from CentOS 7 to be compatible with VFX Platform 2023, and I don't know how feasible that is. Have the big vendors already signalled that their releases next year will be using new ABI and some other Linux distro will be their standard for support?

--

Larry Gritz

l...@larrygritz.com

[Nick Cannon]

On Mon, May 9, 2022 at 8:59 AM Neal Gompa <ngom...@gmail.com> wrote:

This might be a stupid question, but why not upgrade to glibc 2.34
(EL9) rather than glibc 2.28 (EL8)?

It's a good question. Software vendors have many studio customers still using older distributions so it's important for them that we strike a balance between moving the Platform forward without going so far that it cuts off a large portion of their customer base.

There is nothing stopping studios using EL9 or any distro that uses newer versions of glibc. The glibc and gcc versions have the up and down arrows that indicate:

• for systems (or software) providing the library at runtime, versions should be considered minimum version required.
• otherwise, for software building software against the library, versions should be considered highest version allowed

Nick

 

[Nick Cannon]

On Mon, May 9, 2022 at 9:02 AM Larry Gritz <l...@larrygritz.com> wrote:

Developers not on MacOS (e.g., internal development at a Linux-based studio) don't care.

Fair, but software vendors of complex DCCs with a large MacOS customer base may care. If they do then they should let us know.

 

I'm much more concerned with the ABI change, I think that's going to be incredibly disruptive. Though also inevitable. I'm really not sure how to navigate that, since it will mean link-incompatibility with all the standard packages installed with the OS. I can only assume that this means everybody has to make a hard break away from CentOS 7 to be compatible with VFX Platform 2023, and I don't know how feasible that is. Have the big vendors already signalled that their releases next year will be using new ABI and some other Linux distro will be their standard for support?

It's a big change but I believe all major Linux distros made the move to the new ABI long ago. IIRC, RHEL/Rocky/Alma have it as the default since 8.0.

Good questions around the software vendor strategy around Linux support. The community's Linux distro strategy is going to be the main topic of conversation at our SIGGRAPH BoF event this year.

Thanks,

Nick

 

[Neal Gompa]

On Mon, May 9, 2022 at 12:29 PM Nick Cannon <nick....@gmail.com> wrote:
>
> On Mon, May 9, 2022 at 9:02 AM Larry Gritz <l...@larrygritz.com> wrote:
>>
>>
>> Developers not on MacOS (e.g., internal development at a Linux-based studio) don't care.
>
>
> Fair, but software vendors of complex DCCs with a large MacOS customer base may care. If they do then they should let us know.
>
>>
>> I'm much more concerned with the ABI change, I think that's going to be incredibly disruptive. Though also inevitable. I'm really not sure how to navigate that, since it will mean link-incompatibility with all the standard packages installed with the OS. I can only assume that this means everybody has to make a hard break away from CentOS 7 to be compatible with VFX Platform 2023, and I don't know how feasible that is. Have the big vendors already signalled that their releases next year will be using new ABI and some other Linux distro will be their standard for support?
>
>
> It's a big change but I believe all major Linux distros made the move to the new ABI long ago. IIRC, RHEL/Rocky/Alma have it as the default since 8.0.
>

The ABI change happened in 2015, I believe. So it's been a very long time now...

> Good questions around the software vendor strategy around Linux support. The community's Linux distro strategy is going to be the main topic of conversation at our SIGGRAPH BoF event this year.
>

Sadly, as I'm not directly part of the industry, I wouldn't be able to
attend that. I would have loved to otherwise. :(

[Neal Gompa]

In practice, this doesn't seem to be the case. While I with my ISV hat
on am totally fine with that, most ISVs in your industry hard-lock to
specific minor versions of RHEL, even though that has been a bad
practice and heavily discouraged by Red Hat since RHEL 5.

Since you're creating an ABI break with the libstdc++ ABI change, I
think it's worth considering moving to EL9 rather than EL8.

And if you're worried about RHEL clones, it looks like AlmaLinux 9.0
will probably be available very shortly after RHEL 9.0 is, judging by
their release of AlmaLinux 9.0 Beta:
https://almalinux.org/blog/almalinux-9-beta-now-available/

[Larry Gritz]

On May 9, 2022, at 9:29 AM, Nick Cannon <nick....@gmail.com> wrote:

On Mon, May 9, 2022 at 9:02 AM Larry Gritz <l...@larrygritz.com> wrote:

Developers not on MacOS (e.g., internal development at a Linux-based studio) don't care.

Fair, but software vendors of complex DCCs with a large MacOS customer base may care. If they do then they should let us know.

Yeah, yeah. That was what my subsequent paragraph addressed -- those vendors have to do builds/CI on Mac and thus will naturally avoid the features that aren't supported on all the compilers.

--

Larry Gritz

l...@larrygritz.com

[Deke Kincaid]

> Since you're creating an ABI break with the libstdc++ ABI change, I
> think it's worth considering moving to EL9 rather than EL8.

For good or bad, some DCC vendors have already made a decision to update to a RHEL 8 based distribution(Rocky) for their releases that came out this year so far (Autodesk Maya & Flame).

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAEg-Je8ge-ZP%3D_RPSYAPSSNk7oVnVWVmkMWq2N6eQQVF8PLQBQ%40mail.gmail.com.

[Robert Fanner]

@larry, in addition to the points Nick made, my feeling is that an additional niggle we'd need to navigate with C++20 & macOS is how libc++ compatibility is managed when going from "experimental" or partial support for some features to full support.

With libstdc++ on Linux, we've found there to be (understandable) breakages as something matures from experimental to fully supported. The SO post below summarises the gist of the aforementioned more neatly than I could express it. The concern being that libc++ handles things similarly and would manifest similar breakages:

https://stackoverflow.com/a/49119902/601626
"...Similarly, if you compile one object with GCC 7 and -std=c++17 and another object with GCC 8 and -std=c++17 you will have problems, because C++17 support in GCC 7 and 8 is still experimental and evolving."

All the best,

Rob

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAOOm49S4KZCfK-mXMJuf5RT1NPE%2BjsEcV7V5aee%3DC%2BXx1fS2VA%40mail.gmail.com.

--

Rob Fanner

Director of Engineering - Platform

Web: www.foundry.com

The Foundry Visionmongers Ltd.  -  Registered in England and Wales No: 4642027  -  Address: 5 Golden Square, London, W1F 9HT  -  VAT No: 672742224

[Neal Gompa]

On Mon, May 9, 2022 at 3:41 PM Deke Kincaid <dekek...@gmail.com> wrote:
>
> > Since you're creating an ABI break with the libstdc++ ABI change, I
> > think it's worth considering moving to EL9 rather than EL8.
>
> For good or bad, some DCC vendors have already made a decision to update to a RHEL 8 based distribution(Rocky) for their releases that came out this year so far (Autodesk Maya & Flame).
>

Is that a good enough reason to not go *higher* in the 2023 standard?
It's not like that software will suddenly not work on RHEL 9. If they
know that the standard is going to it, they can validate it to be
supported easily enough. But this is also why I'm asking to consider
pushing up, because otherwise they might not bother to add it for a
long time.

[Richard Addison-Wood]

According to this summary:

https://gcc.gnu.org/projects/cxx-status.html

it would appear that C++20 support is still experimental.

---

From: vfx-platfo...@googlegroups.com <vfx-platfo...@googlegroups.com> on behalf of Robert Fanner <robert...@foundry.com>
Sent: May 10, 2022 21:27
To: Larry Gritz <l...@larrygritz.com>
Cc: vfx-platform-discuss <vfx-platfo...@googlegroups.com>
Subject: Re: [vfx-platform-discuss] CY2023 Draft Published

 

*Originates outside of Weta Digital

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAEGe5KA_O%3D11nT-pbW1LuR5yYrVeiteKKVeptnM-e0hnEfh1yA%40mail.gmail.com.

[Larry Gritz]

I suspect that is just left over from when the page was describing pre-C++20 and was never changed. As far as I know, all the stuff in green is fully implemented according to the standard, except where noted.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/SYBP282MB3028CCDE1C1433AC270698F59CC99%40SYBP282MB3028.AUSP282.PROD.OUTLOOK.COM.

--

Larry Gritz

l...@larrygritz.com

[Michael Oliver]

Thoughts on Python 3.11?  It is set to release in October (after SIGGRAPH so timing is not ideal) but the early benchmarks are showing some pretty significant speedups 10-60% faster than Python 3.10.  Blender will be moving to it and I know they Blexit'ed (to quote my British colleague) from  the platform but perhaps this is an opportunity to show that the platform is making an effort to stay current and also ensuring our creatives have the highest performing toolset.

[Neal Gompa]

Well, after dealing with all the crap required to make things happy with Python 3.10 (you'd think having a two-digit minor number wouldn't be a big deal, but nope...) in Fedora, 3.11 should be cakewalk. The performance improvements are extremely significant and certainly worth consideration to promote for v2023.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/c9937b0d-83f1-4d7a-855e-380594e043c9n%40googlegroups.com.

[Nick Cannon]

The VFX Reference Platform schedule requires that all versions for the next calendar year are released by September 1st in order to give software vendors time to migrate, integrate and test before major releases that happen from January to December. We do consider exceptions, as indicated in the CY2023 Draft, for releases as late as 1st November as long as there is agreement from the community. These exceptions are usually the industry-specific libraries that are easier to integrate and test than the more foundational components like Qt or Python.

So, the reason Python 3.10 was selected for CY2023 is that it will be the latest released version by the September 1st cut-off. If people want to make a case for Python to be an exception for CY2023 where we go for 3.11 as long as it releases by November 1st then please make yourself heard either on this thread or privately by emailing feedback at vfxplatform.com. We'd also like to hear from those in the community who would prefer CY2023 sticks with 3.10.

For those that are not already aware, an additional consideration with Python 3.11 is that a number of features are being removed in that release after being deprecated in previous releases:

https://docs.python.org/3.11/whatsnew/3.11.html#removed

Nick

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/c9937b0d-83f1-4d7a-855e-380594e043c9n%40googlegroups.com.

[Edward Lam]

On Mon, May 9, 2022 at 11:24 AM Nick Cannon <nick....@gmail.com> wrote:

The Draft CY2023 VFX Reference Platform has been published at https://vfxplatform.com.

 

On the note to move to oneTBB for CY2024, I'd presume the plan is to also move to oneMKL for CY2024 as well?

Thanks,

-Edward

On Mon, May 9, 2022 at 11:24 AM Nick Cannon <nick....@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CABiBYowY1c9pK_8UqED2vToHZmurH3p_xxuAUNPcP8kBFn%3DJyQ%40mail.gmail.com.

--

Support GOToronto by rating us in the App Store!

[Nick Cannon]

On Fri, May 27, 2022 at 4:14 PM Edward Lam <triplequ...@gmail.com> wrote:

On Mon, May 9, 2022 at 11:24 AM Nick Cannon <nick....@gmail.com> wrote:

The Draft CY2023 VFX Reference Platform has been published at https://vfxplatform.com.

 

On the note to move to oneTBB for CY2024, I'd presume the plan is to also move to oneMKL for CY2024 as well?

Yes, we don't call it out explicitly but CY2024 is the year we plan to migrate both TBB and MKL to the new oneAPI model. A note to that effect will be added next time the site is updated.

Thanks,

Nick

[Alejandro Enriquez]

Hello everyone,

One question, moving forward, will the VFX platform choose between Alma or Rocky or both will be part of the standard? I am mentioning this because Alma 9 brings support for new desktops such as KDE, and having that alternative would be awesome but I see Rocky Linux is having a lot of support.

Jose Enriquez

[Mike Rochefort]

On Sun, Jun 5, 2022 at 5:27 PM Alejandro Enriquez <alen...@gmail.com> wrote:
>
> ... I am mentioning this because Alma 9 brings support for new desktops such as KDE...

The AlmaLinux desktops are provided by EPEL, i.e. the same
project/repo folks have been using to date to install XFCE, Cinnamon,
MATE, etc on CentOS 7. You can see how their Live Media installers
were generated via their Live Media SIG repository.

https://github.com/AlmaLinux/sig-livemedia
https://wiki.almalinux.org/LiveMedia.html
https://docs.fedoraproject.org/en-US/epel/

Live Media is not required for installing and using these desktop
environments on EL8 or EL9.

EL8 and EL9 have the following desktop environments available:

- GNOME 3 / GNOME 40 - Red Hat
- KDE Plasma 5 (maintained by the Fedora KDE SIG) - EPEL
- XFCE 4.16 - EPEL

--
Mike Rochefort
Solution Architect, Red Hat

[Marco Meyer]

By the way: What's the status of that COPR repo for MATE that you've
set up a while back for EL9 (
https://copr.fedorainfracloud.org/coprs/mroche/mate-desktop/)? Are you
planning to keep this around and/or update it for a while, since the
MATE packaging on EPEL has pretty much stalled. Not sure if these
packages were tested and considered production-ready but I'm sure many
VFX shops would appreciate a trustworthy package source :)

[Bob Davis]

To the best of my knowledge, the volunteer maintainer of MATE has stopped maintaining it (which is why it's not in EPEL 8 or 9), but has offered it up to others. Remember - EPEL is a volunteer community, not run by or directed by Red Hat. 

There's still a good bit missing from EPEL 9 as compared to EPEL 8, but the amount of EPEL 9 content out there today is immense compared to where we were with EPEL 8 this soon after the GA of RHEL 8. A long beta period and the existence of CentOS Stream are big reasons for that.

Bob

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/d3ea6ebb4f4b4f3f832626a9d2efb6cd128ac1dc.camel%40marcomeyer-vfx.de.

--

Bob Davis

Principal Product Manager, Red Hat Enterprise Linux

bobd...@redhat.com | 210-452-8945

[Mike Rochefort]

I've written up some thoughts from the Linux distribution perspective about the CY2023 draft that are a bit too long to fit in an email. The document looks at the impact of the distribution touching changes, and my personal suggestions for both software providers and studios as the industry navigates these changes. Any and all feedback would be appreciated!

https://paper.dropbox.com/doc/VFX-Reference-Platform-CY2023-and-Linux-BB1JCtWmYMvFcuElaAokL
For those who have never used Dropbox Paper, if you end up in dark mode and don't like it just hit the question mark in the lower right to find the setting to change the mode. There's also an outline on the left hand side for jumping between sections.

===
@Marco To be honest, that COPR was intended to be little more than a POC for getting MATE started on EL9 to see if it would build easily. I don't plan on revisiting the project or maintaining it as MATE is not something that I personally use. There may be some other COPRs out there looking at MATE on EL9, but I can't vouch for them.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAFj%3DrwOjemqQutLiJBmXfcWfK722p_TteOb90Qw2T%3Daa2-kiwQ%40mail.gmail.com.

[Nick Cannon]

Thank you Mike for putting this thorough and thoughtful document together. Despite your disclaimer, it seems like a well-balanced perspective to me and I encourage anyone interested in the topics of Linux distribution selection or software product support to read it.

To directly address Jose's question...

On Sun, Jun 5, 2022 at 4:27 PM Alejandro Enriquez <alen...@gmail.com> wrote:

One question, moving forward, will the VFX platform choose between Alma or Rocky or both will be part of the standard? 

We will have clearer guidance on this question later this summer, in late July or early August. The VFX Linux Task Force is talking to the major distro providers and software vendors, gathering their research, and will publish a report including a recommendation for which distro the community should align on in a post CentOS Linux world. Mike's document makes the case for specifying distro support more broadly than just a single downstream distribution.

As downstream distributions from RHEL, both Alma and Rocky are "bug for bug compatible" and binary compatible so there they are pretty much indistinguishable from a platform compatibility perspective. 

Nick

[David Aguilar]

Hi Nick,

I'd also be in favor of jumping to Python 3.11 for CY2023.

Indeed, the Python release schedule notes that by September 1st they'll only have a (final) release candidate and that the 3.11 release is planned for October 1st. It basically missed the cutoff by a month, which is a bummer.

https://peps.python.org/pep-0664/

While the removals in 3.11 are a bit of a concern, most of them are in rather esoteric corners of the standard library and hopefully everyone has been using vfx2020, vfx2021 and vfx2022 to adjust their usage of deprecated features.

One of the major problems not present in 3.11 is the upcoming distutils removal. Luckily that hasn't happened in 3.11 -- it's marked deprecated but distutils has not been removed in 3.11. That helps because we won't have to deal with all of the build renovations that will be required when distutils is removed in 3.12.

I agree with Neal G. that the performance wins in 3.11 are noteworthy and might make the pill a little easier to swallow.

Another consideration for me is that we know that CY2023 is going to be a big shift. Notably, the C++ ABI is changing. Thus, my prediction is that this revision of the vfxplatform is one that's going to take a little bit longer for vendors and studios to adopt when compared to previous revisions. And, for that reason, it might make sense to be a little more leading / forward-looking with the Python version if we believe that by the time we get there it's already going to be considered "old".

Just my $.02. I wonder how vendors on this list feel about it, though.

Thanks Nick

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CABiBYoymACJt3xUpQNrMP9wOJnY4hYoLnFOu0qAodeibsB8b8w%40mail.gmail.com.

--

David

[Neal Gompa]

Hey Nick,

With the context of Mike's excellent letter here, I'd like to reiterate that I really believe CY2023 should upgrade to the EL9 base (glibc-2.34) rather than recommending the EL8 base (glibc-2.28). This does not preclude compatibility with existing DCC software that has been built for glibc-2.28, as newer platforms have backward compatibility.

From the perspective of a community contributor and someone working at the platform level, getting things fixed in RHEL 9 is considerably easier than RHEL 8. There's a much more active community process around it, and there are more opportunities for customer (RHEL) and community (CentOS Stream) influence to support the industry's needs.

But most importantly, RHEL 9 has just started its lifecycle, and RHEL 8 is nearly halfway through it. Additionally, for those coming from CentOS Linux 7, CentOS Stream 9 and AlmaLinux 9 are easier migration targets since it will require less automation rework for deployments and management.

Recommending EL9 base makes good sense for maximizing the benefit for studios upgrading now and supporting them for project lifecycles.

--

You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CABiBYoyeotu35ww%3Ds9X3fT2jtr7Jby1t6KMA1PcxDKecAXs%2BVA%40mail.gmail.com.

[Alejandro Enriquez]

Agree with you @Neal Gompa, EL9 seems a better option, studios moving towards EL8 will probably extend until the last minute any change to a new platform, we are talking here almost the end of the decade, by then, EL8  will be really obsolete, and the VFX platform really behind in this matter. Now is a good time to move forward with a jump like this.

[David Aguilar]

+1 from me on allowing for RHEL9, thank you Neal and Alejandro.

FWIW internal discussions at our studio so far have very strongly leaned towards skipping RHEL8 and migrating directly to RHEL9 from our current RHEL7 platform. Adjusting the glibc recommendation to allow for RHEL9 seems like an overall win with no(?) downsides from my perspective (as someone that has their eyes set on RHEL9).

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/071ecb69-70eb-4a9f-a891-5d00006d7ab4n%40googlegroups.com.

--

David

[Bob Davis]

Red Hat product manager chiming in - 

Just as a reminder on lifecycles (since it came up) - RHEL 8 is just over 3 years old, which leaves just under 2 more years of updates and new features before it enters "maintenance" phase - which means that things like security issues and major bugs get fixed, but no new features or hardware enablement is done. Presumably, no downstream rebuilds will go beyond this.

For reference - https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase 

It's also worth noting that, unlike prior releases of RHEL for Workstation, RHEL 9 Workstation is eligible for Extended Update Support, which means you can pick an even numbered release (9.0, 9.2, etc.) and stay on that release for 2 years past the standard life of that point release. This is an additional product, but it's definitely useful for some studios who need a much longer support cycle for their workstations. 

https://access.redhat.com/support/policy/updates/errata#RHEL8_and_9_Life_Cycle 

I only bring this up to provide some specifics around the comments Alejandro and Neal made on where EL 8 is in the lifecycle. 

On the topic of Python 3.11, I don't know when that will find its way into RHEL, so I don't know when 3.11 would be "supported" by us. 

Thanks -

Bob

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/071ecb69-70eb-4a9f-a891-5d00006d7ab4n%40googlegroups.com.

[Neal Gompa]

On Thu, Jun 9, 2022 at 3:10 PM Bob Davis <bod...@redhat.com> wrote:
>
> Red Hat product manager chiming in -
>
> Just as a reminder on lifecycles (since it came up) - RHEL 8 is just over 3 years old, which leaves just under 2 more years of updates and new features before it enters "maintenance" phase - which means that things like security issues and major bugs get fixed, but no new features or hardware enablement is done. Presumably, no downstream rebuilds will go beyond this.
>
> For reference - https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
>
> It's also worth noting that, unlike prior releases of RHEL for Workstation, RHEL 9 Workstation is eligible for Extended Update Support, which means you can pick an even numbered release (9.0, 9.2, etc.) and stay on that release for 2 years past the standard life of that point release. This is an additional product, but it's definitely useful for some studios who need a much longer support cycle for their workstations.
>
> https://access.redhat.com/support/policy/updates/errata#RHEL8_and_9_Life_Cycle
>
> I only bring this up to provide some specifics around the comments Alejandro and Neal made on where EL 8 is in the lifecycle.
>
> On the topic of Python 3.11, I don't know when that will find its way into RHEL, so I don't know when 3.11 would be "supported" by us.
>

From the upstream Fedora perspective, Python 3.11 work has just
started for Fedora Linux 37:
https://fedoraproject.org/wiki/Changes/Python3.11

The only way it'll make it in for RHEL 9.1 is if the stack is added to
CentOS Stream 9 once 3.11rc1 is out in August and tracks for 3.11
final release in October, for RHEL 9.1 in November.

This is likely going to need to be a RHEL customer request so that
it's prioritized and added in time.

[Nick Cannon]

Thank you for all the feedback on CY2023 so far. There is a Working Group meeting coming up in a few days and we will revisit the discussion of whether we could set Python to 3.11 despite its late release relative to the VFX Reference Platform schedule.

On Thu, Jun 9, 2022 at 11:36 AM Neal Gompa <ngom...@gmail.com> wrote:

With the context of Mike's excellent letter here, I'd like to reiterate that I really believe CY2023 should upgrade to the EL9 base (glibc-2.34) rather than recommending the EL8 base (glibc-2.28). This does not preclude compatibility with existing DCC software that has been built for glibc-2.28, as newer platforms have backward compatibility.

Great to hear so much support for studios to move to EL9. CY2023 does not do anything to prevent that. 

The VFX Reference Platform does not (yet?) recommend a specific Linux distribution version. It is a guide for software providers and builders to help the community provide binary compatible software that enable integrated pipelines. Studios are free to use whatever distro they choose, as long as the version of glibc it comes with is not lower than the one specified by the Platform.

The reason for CY2023 to set glibc to 2.28 is to encourage studios away from old unsupported Linux distributions in the 2023-2024 timeframe, while still leaving fairly broad compatibility for the software vendors who have many Linux customers that may not be ready for a distro with a minimum of glibc 2.34 in that timeframe. 

If there are specific features in newer versions of glibc than 2.28 that are important to the community and that may sway this decision then please let us know. The point about getting things fixed in EL9 vs EL8 due to the more active community around it is a great one and we will take that into consideration.

We would love to hear from software vendors who have a strong opinion either way between glibc 2.28 vs 2.34 for CY2023. If you don't feel comfortable replying to this thread then please email the Working Group privately at feedback at vfxplatform.com

In the meantime, the VFX Linux Task Force is actively researching and evaluating Linux distro options for the community and will be publishing a recommendation in August. If there is community support for that recommendation, then that may influence the future of the VFX Reference Platform to be more specific about Linux distributions and versions.

Nick

 

[Brecht Van Lommel]

From the point of view of the Blender project, using Python 3.11 would
be great. The Python version is the only thing keeping us from being
fully compatible, for everything else we match 2022, and 2023 will
probably be the same.

My understanding is that the Python 3.11 API is already stable since
it went into Beta last month, so it's possible to get started with it
now. By itself that's earlier than some other VFX libraries. But the
problem is that not all Python packages will work yet, and it's
difficult to predict the timeline for them. Certainly binary wheels
are likely to only appear after the final release date (though for
Blender we build all packages from source and it generally just
works).

Regarding the glibc version, as a software vendor it's not ideal to
release software that only works on an operating system released ~9
months ago (both for RHEL and Ubuntu LTS). Certainly on Windows or
macOS that would be unheard of. For Blender we'll probably build on
either RHEL/Alma/Rocky 8 or perhaps Ubuntu 18.04, since it would be
nice to keep supporting that too. From our point of view, if the VFX
platform further raises the glibc version that's fine, as we can
target lower glibc and remain compatible. Maybe there would be some
indirect issues down the line for library dependencies or toolkits
like CUDA or OneAPI dropping support for older distributions, though
there are more industries than VFX affecting such decisions.

> --
> You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CABiBYozXn9uMHrBbBOOjgGN5q884FHDGbh%2Bm4ytZMWKuKcGGHg%40mail.gmail.com.

[Neal Gompa]

On Mon, Jun 13, 2022 at 8:34 AM Brecht Van Lommel
<brechtv...@gmail.com> wrote:
>
> From the point of view of the Blender project, using Python 3.11 would
> be great. The Python version is the only thing keeping us from being
> fully compatible, for everything else we match 2022, and 2023 will
> probably be the same.
>
> My understanding is that the Python 3.11 API is already stable since
> it went into Beta last month, so it's possible to get started with it
> now. By itself that's earlier than some other VFX libraries. But the
> problem is that not all Python packages will work yet, and it's
> difficult to predict the timeline for them. Certainly binary wheels
> are likely to only appear after the final release date (though for
> Blender we build all packages from source and it generally just
> works).
>

Interpreter stability isn't guaranteed until RC. That's what *really* matters.

> Regarding the glibc version, as a software vendor it's not ideal to
> release software that only works on an operating system released ~9
> months ago (both for RHEL and Ubuntu LTS). Certainly on Windows or
> macOS that would be unheard of. For Blender we'll probably build on
> either RHEL/Alma/Rocky 8 or perhaps Ubuntu 18.04, since it would be
> nice to keep supporting that too. From our point of view, if the VFX
> platform further raises the glibc version that's fine, as we can
> target lower glibc and remain compatible. Maybe there would be some
> indirect issues down the line for library dependencies or toolkits
> like CUDA or OneAPI dropping support for older distributions, though
> there are more industries than VFX affecting such decisions.
>

It is increasingly normal on the macOS side, especially with the new
ARM-based Macs resetting expectations again, and Microsoft is driving
towards a similar mentality with Windows 11. I would be extremely
surprised if we wouldn't start seeing software raising their minimums
pretty aggressively. Especially with Microsoft building in new OS
functionality exclusively in Windows 11 that toolkits would take
advantage of. As it is, the VFX platform for macOS already has a
minimum of macOS 11, so in that respect, it's already aggressive.

The key advantage of raising the minimum is to maximize the lifecycle
that ISVs, IHVs, and studios have for ensuring ecosystem development
can be prioritized and released to benefit everyone.

[Brecht Van Lommel]

On Mon, Jun 13, 2022 at 3:23 PM Neal Gompa <ngom...@gmail.com> wrote:
> Interpreter stability isn't guaranteed until RC. That's what *really* matters.

From my point of view, the most important is API stability so we can
start development and testing early. If there's a bug in the
interpreter, it's a good thing we can report that before it's too
close to the final release.

> It is increasingly normal on the macOS side, especially with the new
> ARM-based Macs resetting expectations again, and Microsoft is driving
> towards a similar mentality with Windows 11. I would be extremely
> surprised if we wouldn't start seeing software raising their minimums
> pretty aggressively. Especially with Microsoft building in new OS
> functionality exclusively in Windows 11 that toolkits would take
> advantage of. As it is, the VFX platform for macOS already has a
> minimum of macOS 11, so in that respect, it's already aggressive.

I can't predict the future, but dropping support for even 10% of users
would be a big ask, while I'd guess it would be easily 50% if we
require Windows 11 and RHEL 9 next year. It just seems far removed
from the practical reality today.

Anyway, I'm not trying to make an argument for or against a particular
Linux glibc version since either works for us as a software vendor,
just giving background on what Blender is likely to do.

[Neal Gompa]

On Mon, Jun 13, 2022 at 2:23 PM Brecht Van Lommel
<brechtv...@gmail.com> wrote:
>
> On Mon, Jun 13, 2022 at 3:23 PM Neal Gompa <ngom...@gmail.com> wrote:
> > Interpreter stability isn't guaranteed until RC. That's what *really* matters.
>
> From my point of view, the most important is API stability so we can
> start development and testing early. If there's a bug in the
> interpreter, it's a good thing we can report that before it's too
> close to the final release.
>

Interpreter stability is required for language bindings to be made
properly for a language. That's especially important for PyQt and PySide.

> > It is increasingly normal on the macOS side, especially with the new
> > ARM-based Macs resetting expectations again, and Microsoft is driving
> > towards a similar mentality with Windows 11. I would be extremely
> > surprised if we wouldn't start seeing software raising their minimums
> > pretty aggressively. Especially with Microsoft building in new OS
> > functionality exclusively in Windows 11 that toolkits would take
> > advantage of. As it is, the VFX platform for macOS already has a
> > minimum of macOS 11, so in that respect, it's already aggressive.
>
> I can't predict the future, but dropping support for even 10% of users
> would be a big ask, while I'd guess it would be easily 50% if we
> require Windows 11 and RHEL 9 next year. It just seems far removed
> from the practical reality today.
>
> Anyway, I'm not trying to make an argument for or against a particular
> Linux glibc version since either works for us as a software vendor,
> just giving background on what Blender is likely to do.
>
>

The biggest advantage of glibc-2.34 is that hwcaps and hw accelerated
libraries based on CPU capabilities are available. This makes it
considerably easier to ship high performance libraries with graceful
fallbacks.

> > The key advantage of raising the minimum is to maximize the lifecycle
> > that ISVs, IHVs, and studios have for ensuring ecosystem development
> > can be prioritized and released to benefit everyone.
>

> --
> You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAKFUgC01aw8TeCjCX%2B1MSjxpgL66p6PC8Dy8kav_EV2dd5kHEg%40mail.gmail.com.

[Larry Gritz]

Aside: is there a reference handy explaining how to take advantage of this?

On Jun 13, 2022, at 12:10 PM, Neal Gompa <ngom...@gmail.com> wrote:

The biggest advantage of glibc-2.34 is that hwcaps and hw accelerated
libraries based on CPU capabilities are available. This makes it
considerably easier to ship high performance libraries with graceful
fallbacks.

--

Larry Gritz

l...@larrygritz.com

[Neal Gompa]

I don't have a specific official reference, but the details of how it
works are documented in the patch set that was merged for glibc 2.33:
https://sourceware.org/pipermail/libc-alpha/2020-June/115250.html

It's briefly mentioned in a blog post about RHEL 9 too:
https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level

[Edward Lam]

On Mon, Jun 13, 2022 at 3:29 PM Neal Gompa <ngom...@gmail.com> wrote:

It's briefly mentioned in a blog post about RHEL 9 too:
https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level

This is interesting, I just noticed that RHEL 9 has the minimum x86-64 processor requirements:

• x86-64-v2 brings support (among other things) for vector instructions up to Streaming SIMD Extensions 4.2 (SSE4.2)  and Supplemental Streaming SIMD Extensions 3 (SSSE3), the POPCNT instruction (useful for data analysis and bit-fiddling in some data structures), and CMPXCHG16B (a two-word compare-and-swap instruction useful for concurrent algorithms).

I don't know if any studios still have older hardware but this would certainly be a consideration for RHEL 9. FWIW, and mind you this was 4 years ago, we were not able to upgrade our minimum requirements to SSE4.2 and had to go back down to SSE4.1 due to render farm machines of one particular large client.

-Edward

[Neal Gompa]

I would be surprised if you have that problem today, since x86_64-v2 has been implemented in most x86_64 processors since 2010, and definitely in all since 2015.

[Deke Kincaid]

Many (most?) software renderers implemented that requirement a while ago.  Going through my emails it looks like Arnold did this in 2016, Vray 2018, Houdini/Mantra 2018, Renderman 2019, etc...

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAJZe6MAeNigheWn0dG6xpdLVrfOAP%3DKLrfpff%2BO8iHGWyLwqkQ%40mail.gmail.com.

[Edward Lam]

On Thu, Jun 16, 2022 at 6:20 PM Deke Kincaid <dekek...@gmail.com> wrote:

Many (most?) software renderers implemented that requirement a while ago.  Going through my emails it looks like Arnold did this in 2016, Vray 2018, Houdini/Mantra 2018, Renderman 2019, etc...

For the record, Houdini/Mantra is exactly what I was talking about, which backed down to SSE 4.1 as the minimum requirement. However, its VEX JIT compiler (for shaders) is able to take advantage of newer instructions at run time (of course).

Best regards,

-Edward

PS. Sorry for the re-post as I accidentally replied privately before.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAOOm49SGpvqTkFkDXoKUZ9ZBgMdWtUsQP5id%2B0HW%3Dqg0W2UCpA%40mail.gmail.com.

[Ashley Whetter]

I don't know if this has been discussed already but Python now has a set annual release cycle (https://peps.python.org/pep-0602/). So because the VFX platform is defined annually, theoretically we could set the expectation that any VFX platform version will use either the version of Python that enters its RC phase in August that year and is released in October, or the platform version will use the earlier version of Python that will have been released in October the year prior.

As a user I think latest is greatest, but I don't have enough knowledge of the complexities of the VFX platform to really advocate one way or the other. I'm simply putting this out there as an option to consider (perhaps from the next platform version given that this version is already in draft).

Ashley

[Nick Cannon]

The annual release cycle for Python is definitely helpful and could offer a path to more predictable adoption of new versions. 

In the case of Python 3.11, it unfortunately seems like there is a possible delay in releasing it until December which would be too late for CY2023. 

https://mail.python.org/archives/list/pytho...@python.org/thread/3JWVCSBPBFWY5ZWSJ7RYB6FS5NIMCEOY/

We'll keep a close eye on progress and I'm sure this will be a topic of discussion at our SIGGRAPH BoF.

Nick

[Bob Davis]

In regard to Python, even if the upstream version does come out on time, there's still delay in having it made available via the distros.

If it's important for Python to be built in and "supported" by the linux distro, that matters. If upstream Python from the PSF is acceptable, this is less of a problem. 

Bob

--

You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CABiBYoxy9ivjtmtk_M8HjZnKa%2Bsv29PyPZtPb6Wfi9miDmxN%3Dw%40mail.gmail.com.

[Larry Gritz]

And there's also a delay in the apps themselves and other development tasks. Just because a dependency is released by New Years Day doesn't mean that everything is fine for downstream packages being released for the new calendar year. The "BigDCC 2023" release might be in customers' hands as a beta or preview as early as the first half of 2022, and it's in development for a significant time before that. That development and testing needs all the major dependencies to be available.

And it's a whole web of dependencies. We don't just need the new python, we need the new pybind11 and boost.python releases that are known to work with the new python. And we need the libraries (like OTIO, etc.) that are known to work with the new python and pybind11. And so on. Support has to make is way through every link in the chain of the software stack before we can throw the switch on a new version.

The October deadline (and strong preference for a final or near-final version by SIGGRAPH at the latest) seems not just reasonable, but actually fairly generous. (Though there is considerable package-by-package variation in how much lead time the downstream products will need to have access to a near-final version of it; some are easier than others to swap in updated versions.)

That all said, a strong argument can be made for always staying at most one year behind, giving devs and product testing plenty of time with a released version, while not letting ourselves get into another situation where we are behind by several major releases. I continue to say that lagging as much as we sometimes do, we leave a huge pile of money on the table each year -- in the form of not having performance improvements, new features, and developer productivity that are actually available. We can be wary of depending on releases that are just in the nick of time, while still trying much harder to be on the latest established releases of important tools.

-- lg

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAFj%3DrwNrNQ2Ym5wQqO9L5J9FW2GDL8cTva6CBLNiTUwxEVcj_g%40mail.gmail.com.

--

Larry Gritz

l...@larrygritz.com

[Neal Gompa]

I think it's fairly reasonable to stick with Python 3.10 right now,
all things considered.

Unlike 3.11, the integration work is done, and for RHEL, it's just a
matter of pulling it into RHEL 9 now once a customer asks for it (or
if it's already on the schedule, it will probably show up soon).

> To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/441FC407-B962-4D18-8833-54B16BFC4B68%40larrygritz.com.