Python 3.7 on CentOS 7 (and 8)

[Anthony Nemoff]

Hi,

I'm currently working on adding support for Python 3.7 on Isotropix Clarisse.

On Windows and macOS, the official Python installer is easy to download and use. But on CentOS 7, official Python binaries in repositories go up to 3.6, not 3.7. So to install 3.7, you need to build it.

Maybe I'm wrong, if so, I'd like to know how to get Python 3.7 binaries out of the box on CentOS 7.

For information, we are going to support Python 2 and 3 on our next major version, configurable by a command line option or environment variable. Then we will drop Python 2 when when next+1 major version comes out.

We don't embed Python binaries in our application. So far we have relied on Python being present in the system or properly installed by the user. For Python 2.7 this was ok as it's the system's default version.

But for Python 3.7, I don't know how CentOS 7-8 (or RHEL) users can easily install Python 3.7 binaries. Asking them to manually download, build and deploy it is not a friendly user experience.

I was curious how other applications are dealing with this? Are you supporting only Python 3.6 to rely on system capabilities? Are you embedding 3.7 in your application files?

I don't know if this has been discussed before, but I was curious why Python 3.7 was chosen instead of 3.6 knowing that CentOS 7 and 8 don't provide 3.7 out of the box yet.

I'm in favor of moving to newer versions and know that 3.6 is going EOL at the end of 2021, but in this case the required Python version isn't available on CentOS or RHEL distribs currently used in VFX studios.

Thanks!

-Anthony

[Charles Fleche]

Most other applications ship their own python interpreter and libs, and that can cause issues as they don't namespace their modules properly, or at all, or ship python in slightly un-standard ways. It'd definitely be much easier to rely on the the system provided python installation, for sure.

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/e409ff6a-87e1-4e20-82f3-daac615ed78b%40googlegroups.com.

--

Charles Flèche  R&D Software Developer cfl...@rodeofx.com  www.rodeofx.com

[Anthony Nemoff]

Thanks, that's good to know.

Relying on system's Python has its advantages when properly installed. But when it's not the case (custom install or whatever), it can also lead to other problems. But those problems are usually easy to fix in most cases by fixing the environment variables for example. For less technical users it usually requires more explanations, but still fixable.

I'm curious to get feedback from other developers about their experience with embedding Python in their application files.

[Narendra Kumar Vadapalli]

Packing a compiled python interpreter with the application (Maya) has just causing me pain since last 10 days. If you leave the Python out of the application, it makes a lot of things easier. Glad ot see that at least some application is not doing it.

On the other hand, yeah CentOS plan for getting to Python 3.7 is still not known ! But I see attempts of the same happening when searched on Internet (https://tecadmin.net/install-python-3-7-on-centos/)

~~~~~~~~~~~~~~~~~

Best Regards,

Narendra Kumar Vadapalli,

Sr Software Developer at Animation Studio

https://www.narenvadapalli.com/

~~~~~~~~~~~~~~~~~~~

--

[Aloys Baillet]

Hi Anthony,

Here is the script that installs python (2.7 and 3.7) on CentOS 7 docker images for our VFX platform CI: https://github.com/AcademySoftwareFoundation/aswf-docker/blob/master/scripts/base/build_python.sh

That should help you.

Cheers,

Aloys

[Chad Dombrova]

I'm curious about the problems people have had with applications that provide their own interpreter?  I can't recall ever having any issues.  

-chad

 

[Anthony Nemoff]

Thanks for the feedback.

I had already built 3.7 myself, that was not an issue. Thanks for the links anyway.

The issue is that this is not a good workflow for end users: we don't want end users to build and deploy Python themselves, even with the help of a script.

To be honest, not embedding Python in our application has been a mixed experience until now. When Python 2.7 was present it was transparent for the user.

But when it wasn't, it did generate a lot of support requests, mostly from indies or small studios that don't have IT guys to solve these kind of issues. For example it was very common, on Windows, that users skipped/forgot to install Python (even if we embed the Python installer), or forget to add it to PATH.

In the past we've thought on embedding it to resolve such issues and provide a nicer "first run" experience of our app. We are thinking on doing it now with the migration to Python 3, mostly due to the fact that 3.7 binaries are not a available out of the box on some linux distros. Also, vfxplatform Python requirements might move faster to newer Python version than linux distros do.

Therefore I am also curious about the problems people had with embedded Python binaries in other applications. I can imagine it can be related to mismatching Python versions and modules.

Cheers,

-Anthony

[Alan Fregtman]

> I'm curious about the problems people have had with applications that provide their own interpreter?

My most recent pain in this regard was when Shotgun deprecated support for very old (insecure) TLS 1.0 and 1.1 SSL encryption, thus breaking the Python Shotgun API module in Nuke 10 and earlier due to their embedded Python being an old compile whose libssl wasn't new enough to know of TLS 1.2+ SSL standards.

We had to set up a local proxy just to patch through the SG API calls without SSL until productions on Nuke 10 finally ended and we could get rid of it.

On that note, maybe "libssl minimum version" ought to be VFX-Platform-tracked in this day and age where encryption is everywhere?? 🤔 I don't see it in the list. (It'd be nice if all bundled Pythons had the same modern SSL libs.)

On Wed, Feb 19, 2020, 1:06 AM Chad Dombrova <cha...@gmail.com> wrote:

I'm curious about the problems people have had with applications that provide their own interpreter?  I can't recall ever having any issues.  

-chad

 

--

You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAGq9Q7Ecv8b8LJxbDwXvnb7AfJ6-ttvaZmhR2RBUhfZoVMHCrQ%40mail.gmail.com.

[Narendra Kumar Vadapalli]

Hi Chad,

For example, I wanted to build grpcio python library with Maya 2019, which comes with a custom python interpreter that was compiled with MSVC 1900 (Visual Studio 2015).  But Maya doesn't support creation of Virtual environments, even if you were able to install setuptools and virtualenv in to mayapy.  I had to do all kinds of hackery to get the build working since Maya bundles Python in an unconventional way.

• All the site packages etc... are packaged under a zip file under bin folder called python27.zip
• The Include folder which is supposed to hold Python.h header file necessary for any compilation of C extensions using Cython is missing and that is found under libs/Python2.7 (or similar, don't recollect exactly)

If the python interpreter shipped with Maya had an alternative executive file called python.exe (which could be a symlink to mayapy.exe) with the folders in expected locations a.k.a follow the structure of a regular shipped python interpreter, I am fine with packaging the interpreter along with the application

I might end up writing  blog with exact details.. Will post it here when I get to it

~~~~~~~~~~~~~~~~~~

Best Regards,

Narendra Kumar Vadapalli,

Sr Software Developer at Animation Studio

https://www.narenvadapalli.com/

~~~~~~~~~~~~~~~~~~~

On Tue, Feb 18, 2020 at 10:06 PM Chad Dombrova <cha...@gmail.com> wrote:

I'm curious about the problems people have had with applications that provide their own interpreter?  I can't recall ever having any issues.  

-chad

 

--

[Narendra Kumar Vadapalli]

... Oh forgot to add the details about the pain point of building Python-2.7.11 with MSVC 1900 as there is absolutely zero documentation with few github patches sprinkled available in CPython official repository. Why do you need to do that?

If you don't want to mess up your Maya 2019 installation but to recreate a Python-2.7.11 that can match the exact MS build tools version, then you can happily create virtual environments and do all necessary building of python libraries. So that was my initial attempt and at somepoint, just gave up and altered the Maya's installation to match the dependencies necessary for compilation of the grpcio library.

Btw.. its worth mentioning that my attempt was to do all this on Windows 10 

~~~~~~~~~~~~~~~~~~

Best Regards,

Narendra Kumar Vadapalli,

Sr Software Developer at Animation Studio

https://www.narenvadapalli.com/

~~~~~~~~~~~~~~~~~~~

[Charles Fleche]

What Narendra said: Python is weirdly packaged in Maya. We also had trouble with our Python 2 / 3 migration in Nuke. Using `six` for 2/3 compatibility, as The Foundry is shipping a very old version of `six` if bite us for the code that can be imported within nuke as dependencies, but also in other DCCs.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CADnYgak0igOYigWV%3D0JEQwJSL1pqoCzUpw3eGQKQVDEdQjNSiw%40mail.gmail.com.

[Attila Szabo]

3ds Max also packages Python.

 

Python 2.7 binaries compiled with a recent compiler (ex. VS 2017) were not available, so we had no choice but to rebuild it with VS 2017.

The next version of 3ds Max (2020+1) will ship with off-the shelve Python 3.7 as the default (will also support Python 2.7). This means that python extensions don’t need to be rebuilt (simply pip install). Oh, and we support Python 3 virtual environments!

 

If anyone here wants to check it out and give us feedback, please fill out an application to join our beta program. We would greatly appreciate any feedback.

 

 

Attila Szabo

Product Owner, 3^rd Party Developer Experience

Autodesk 3ds Max

 

Autodesk, Inc.

10 Duke Street, Montreal, QC. H8T 3R4, Canada

www.autodesk.com

Autodesk Show Reel 2019

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAOLWtTSXOTnaveaUB6hc4jmY1k%3DFqONGCi1ef5SJh%3DqisuR-ZA%40mail.gmail.com.

[David Aguilar]

In case anyone's curious, our approach for Python in Maya (on Linux / RHEL7) is that we've always forcibly removed the python libraries from Maya's installation so that finds and uses the system/studio-provided python.

It would be nice if this was an officially separable component so that we could compile our own maya/python bridge, and then provide our own compatible builds of pyside2/openexr/alembic/etc. python modules that maya needs.

We do the same for Maya's shipped Qt and TBB libraries -- we remove them and arrange to have it find our studio installation.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/SN6P136MB006405E14A180BE425FFB49EED100%40SN6P136MB0064.NAMP136.PROD.OUTLOOK.COM.

--

David

[Alan Fregtman]

> What Narendra said: Python is weirdly packaged in Maya. We also had trouble with our Python 2 / 3 migration in Nuke. Using `six` for 2/3 compatibility, as The Foundry is shipping a very old version of `six` if bite us for the code that can be imported within nuke as dependencies, but also in other DCCs.

I got bitten similarly at my current studio by our Thinkbox Deadline install including its own ancient "shotgun_api3" which it would push into the PYTHONPATH by default.

For the DCC core devs reading this thread, if you must include Python dependencies, pretty please... consider doing a "vendor" mechanism so you can include them without affecting normal imports for everyone else.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAOLWtTSXOTnaveaUB6hc4jmY1k%3DFqONGCi1ef5SJh%3DqisuR-ZA%40mail.gmail.com.

[Aloys Baillet]

We actually do something very similar to Disney as we delete python from the maya install, but we replace it with symlinks to avoid breaking some things with mayapy (which is still quite broken for us anyway...).

This works well and allows us to be on the very latest patch of python-2.7.

We also delete Qt, PySide, TBB, Alembic, etc... as we build these from source and have patches to some of them, all resolved with rez.

It's true that python-3 on CentOS-7 is quite a bad out of the box experience indeed, I believe people have raised the desire to move to C++17 for the VFX platform 2021, which in turn might require the DTS from RHEL8, and maybe that would be a good thing... still too late for this year though.

[Anthony Nemoff]

If I understand correctly this page https://access.redhat.com/support/policy/updates/rhscl-rhel7, I deduce that Python 3.7 might become available on CentOS 7 (and likely on CentOS 8 too) sometime this year,

It says that Python 3.6 will be retired on October 2020 (Python 3.5 was retired on May 2019). So I assume 3.7 will be added to the SCL before 3.6 becomes retired.

From the multiple responses, Python 3.7 not being available out of the box doesn't seem to be a problem as most studios working on Linux build it themselves to ensure a consistent environment across the various applications.

[Alan Fregtman]

> It's true that python-3 on CentOS-7 is quite a bad out of the box

Has the VFX Platform committee considered providing a yum repo with good rpms other studios could rely on? It could be a nice way to alleviate this.

It might not work for everybody but could still help many.

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/f9760ec1-d2ab-4ab4-bc11-db0a192883e1%40googlegroups.com.

[Paul Miller]

On 2/21/20 6:28 AM, Alan Fregtman wrote:

> It's true that python-3 on CentOS-7 is quite a bad out of the box

Has the VFX Platform committee considered providing a yum repo with good rpms other studios could rely on? It could be a nice way to alleviate this.

It might not work for everybody but could still help many.

I'd still love to see an official VFX Platform dev distro or container that I could install and build against.

On Fri, Feb 21, 2020, 6:30 AM Aloys Baillet <aloys....@gmail.com> wrote:

We actually do something very similar to Disney as we delete python from the maya install, but we replace it with symlinks to avoid breaking some things with mayapy (which is still quite broken for us anyway...).

This works well and allows us to be on the very latest patch of python-2.7.

We also delete Qt, PySide, TBB, Alembic, etc... as we build these from source and have patches to some of them, all resolved with rez.

It's true that python-3 on CentOS-7 is quite a bad out of the box experience indeed, I believe people have raised the desire to move to C++17 for the VFX platform 2021, which in turn might require the DTS from RHEL8, and maybe that would be a good thing... still too late for this year though.

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/f9760ec1-d2ab-4ab4-bc11-db0a192883e1%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CAN%3DPNLHBtu05m4gjUFOdNEKZmx4V%2Ba0qPSwTd-yUn2LKDwBiEQ%40mail.gmail.com.

[Aloys Baillet]

Maintaining a full set of RPMs for all VFX platform packages sounds great but would be quite a large amount of work and would actually distract from pushing the VFX platform to more OSes...

The current approach of the ASWF is to provide docker images that contain many of the VFX platform packages. This enables studios, vendors and OSS contributors to rely on a Continuous Integration platform to keep delivering great software to the community.

One of the ASWF goals is to also provide this infrastructure to windows and mac, which docker does support, but RPMs do not, so it makes sense to push on with better docker solutions that can be rolled out to these OSes.

Unfortunately at this stage people can't readily run maya from a docker container (you technically can but it's quite some work to setup), so that means these docker images do not currently bring a lot of value to end users.

That said I'm quite confident this will change in the next few years as virtual workstations start becoming more widespread.

I believe a link to the ASWF docker repo was recently added on vfxplatform.com, but here it is: https://github.com/AcademySoftwareFoundation/aswf-docker

Cheers,

Aloys

[Nick Cannon]

On Fri, Feb 21, 2020 at 6:16 AM Paul Miller <pa...@fxtech.com> wrote:

On 2/21/20 6:28 AM, Alan Fregtman wrote:

> It's true that python-3 on CentOS-7 is quite a bad out of the box

Has the VFX Platform committee considered providing a yum repo with good rpms other studios could rely on? It could be a nice way to alleviate this.

It might not work for everybody but could still help many.

I'd still love to see an official VFX Platform dev distro or container that I could install and build against.

We have certainly considered providing these but it's a significant amount of work to create and maintain a distro or high quality RPM packages. Instead we are partnering with the ASWF and very supportive of Aloys and others who are doing great work in creating an automated build/CI system and providing containers for various VFX Reference Platform components.

Our next priority for the Platform is to extend support to Windows and MacOS so our focus is on that for now.

Nick

[Paul Miller]

That is interesting. I haven't done anything with containers yet, but these should prove useful.

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CABiBYoy8QpN3DaiYyWQiyGr%2BV932iob0fDe%2BjeSkJYzo9zpeTg%40mail.gmail.com.

[Aloys Baillet]

Hi again,

There's an interesting trick that you can use to download individual VFX platform packages on linux (if you have docker installed):

docker pull aswf/ci-package-openexr:2019
docker_id=`docker create aswf/ci-package-openexr:2019 null`
docker export --output=/tmp/openexr-2019.tar $docker_id
docker rm $docker_id

will create a uncompressed tar file that contains the OpenEXR library as compiled and used by other aswf ci images.

You can see a list of all the available packages there: https://hub.docker.com/search?q=aswf%2Fci-package&type=image but there are quite a few now, including TBB, Boost, Python, Qt, PySide and even USD-19.11.

I just realised that building RPMs on top of these could actually be quite doable, so I've added a Google Summer of Code idea there https://github.com/AcademySoftwareFoundation/tac/tree/master/gsoc#cross-project-infrastructure to potentially implement.

That said I'll repeat that this is not quite the current focus of both VFX Platform or ASWF at this stage. It would just be a nice bonus...

Cheers,

Aloys

[Anthony Nemoff]

Thanks for sharing.