CY2018 compiler changed to gcc 6.3.1

[Nick Cannon]

As proposed a few weeks ago, CY2018 has now been updated to reflect a late change in the compiler to gcc 6.3.1 due to issues found with gcc5. It is keeping the same legacy ABI setting of _GLIBCXX_USE_CXX11_ABI=0. 

Redhat Developer Toolset 6.1 provides a straightforward way to install gcc 6.3.1 for Redhat and CentOS users.

Thank you for all the feedback. We did consider how to define binary compatibility more broadly than just specifying one specific compiler version but it's a complex issue and we still need more feedback before being able to make such a change. If this is something you feel strongly about then we welcome proposals on how to tackle it, particularly from anyone who has had success creating compatible builds with other compilers.

Nick

[Larry Gritz]

Nick, is that ABI setting the default for gcc 6.3? Or does everybody need to add that definition to every compile in order to avoid problems?

--

Larry Gritz

l...@larrygritz.com

[Nick Cannon]

On 26 November 2017 at 17:12, Larry Gritz <l...@larrygritz.com> wrote:

Nick, is that ABI setting the default for gcc 6.3? Or does everybody need to add that definition to every compile in order to avoid problems?

Good question, like with gcc 5 I believe the default is the new ABI but some distributions set the default to the old.

From https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html:

Using the default configuration options for GCC the default value of the macro is 1 which causes the new ABI to be active, so to use the old ABI you must explicitly define the macro to 0 before including any library headers. (Be aware that some GNU/Linux distributions configure GCC 5 differently so that the default value of the macro is 0 and users must define it to 1 to enable the new ABI.)

To be on the safe side yes, everybody can set _GLIBCXX_USE_CXX11_ABI=0 on every compile in order to avoid problems.

This is based on the documentation so if anyone knows better from direct experience then please reply here to update us all.

Nick

[Richard Addison-Wood]

For what it is worth, you ask a particular installation of gcc how it was configured by doing:

g++ -v

and looking the options that it reports were used for configure.

If you see:

--with-default-libstdcxx-abi=gcc4-compatible

that means that the default would be _GLIBCXX_USE_CXX11_ABI=0.  (When I looked at what is provided via devtoolset-6, that is what is reported.)

On the other hand, if you see:

--with-default-libstdcxx-abi=new

that means that the default would be _GLIBCXX_USE_CXX11_ABI=1.

Given that this configuration will very among distributions, it would indeed be a good idea use -D_GLIBCXX_USE_CXX11_ABI=0 explicitly for code being built for use with the VFX Reference Platform CY2018.

Note that the libraries provided for a particular Linux distribution may be built with the other choice for _GLIBCXX_USE_CXX11_ABI.  However, this had already been something that some facilities would have been encountering depending on their choice of Linux distribution.

--
You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
To post to this group, send email to vfx-platfo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/CABiBYowvzUuuAhAaiLEy%3DoC321%3DAh6gcDn%3DC0qU0hqSuRYuavQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Aloys Baillet]

Hi Nick,

We are finally catching up to 2018 and trying to find that mythical GCC-6.3.1 but no luck so far...

6.3.1 isn't an official GCC version. The best reference to it we could find came from a red hat mailing list where a red hat developer stated that 6.3.1 referred to any snapshot between 6.3 and 6.4. Those snapshots are deleted after a period of time and there doesn't appear to be a version that matches the example given in that mailing list.

So we are guessing we will need to find out which snapshot is the correct one for the recommended redhat developer tools.

Any help you could provide for this quest would be greatly appreciated!

I would also recommend adding more information on the vfxplatform.com website to help find the correct gcc version once it is found.

As a side note, being on CentOS-7 means that installing the Red Had Developer Toolset is not quite straightforward, I found this repo https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/ which might be OK but I can't see any guarantee that the installed toolset would be a perfect match with the toolset developed by Red Hat. We usually prefer building all our packages from source (maintained in rez packages) and we would probably not use the devtoolset anyway unless we need to investigate strange issues...

Cheers,

Aloys

[Wayne Arnold]

to install Red Hat Developer Toolset 6.1 on CentOS 7

 

sudo yum -y install centos-release-scl

sudo yum -y install devtoolset-6

 

--

Wayne

--

You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
To post to this group, send email to vfx-platfo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/c81e75a2-d9a2-42a4-8b6b-cd0cac3e1924%40googlegroups.com.

[Aloys Baillet]

Thanks Wayne! That's fairly easy indeed.

Though I'm still in search of the actual source of the gcc included there...

[Wayne Arnold]

the CentOS source files can be found on the CentOS Vault link

 

http://vault.centos.org/7.5.1804/

 

http://vault.centos.org/7.5.1804/sclo/Source/rh/devtoolset-6/

--

You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
To post to this group, send email to vfx-platfo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/86d4191f-08ae-4b3e-88f0-e7bb38742a4b%40googlegroups.com.

[olivier...@gmail.com]

Le jeudi 21 juin 2018 21:18:07 UTC-4, Aloys Baillet a écrit :
> 6.3.1 isn't an official GCC version. The best reference to it we could find came from a red hat mailing list where a red hat developer stated that 6.3.1 referred to any snapshot between 6.3 and 6.4. Those snapshots are deleted after a period of time and there doesn't appear to be a version that matches the example given in that mailing list.
>

Yeah, it's quite hard to get a hold of. I got mine here: https://src.fedoraproject.org/repo/pkgs/gcc/gcc-6.3.1-20170216.tar.bz2/sha512/0b589f16af671cf255035838350f657ae112774351a5d05a513a29f4ab6be85c18208a1ff9edf85cab8fa05555aea8c01e35647d3993a9e085c0347f4a675a21/gcc-6.3.1-20170216.tar.bz2

not quite sure how I manufactured that URL anymore. I think it was in some other file.

> As a side note, being on CentOS-7 means that installing the Red Had Developer Toolset is not quite straightforward, I found this repo https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/ which might be OK but I can't see any guarantee that the installed toolset would be a perfect match with the toolset developed by Red Hat. We usually prefer

As far as I know, the patches needed to build the special libstdc++ used by the devtoolset are maintained by Red Hat and only exist publicly for very specific versions of gcc. They're a large dump of assembly generated by some process I don't really want to know about. So I think it's a safe bet that it will be the same and be ABI compatible. I don't know of any way to build any other version and believe me, I tried quite hard.

Olivier

[Aloys Baillet]

Thanks a lot Olivier and Wayne!

The links to sources are very useful indeed, and I've got a local build of gcc-6.3.1 finally!

Cheers,

Aloys

[Aloys Baillet]

Hi,

Resurrecting this old(ish) thread as I noticed some interesting (as in scary) things with our custom build of GCC-6.3.1.

I had not realised that the devtoolset-7.1 is managing some kind of engineering feat by allowing GCC-6.3.1 with C++14 enabled to produce executable that are linked to a libstdc++.6.0.19 which is the version of the library produced by GCC-4.8.3(or 4.8.5?) and ships as the default libstdc++ on CentOS-7. This is really neat as it allows vendors to use newer compilers and C++ features without having to ship a new libstdc++!

However when compiling the gcc source of the devtoolset-7.1 I get a new libstdc++.so.6.0.22 which contains newer symbols (such as the new C++14 sized delete operators). We had been running new maya and houdini versions with this libstdc++.so.6.0.22 in the LD_LIBRARY_PATH and everything seemed fine...

...until today :-)

As we've been progressing and migrating more libraries to the new compiler I ran against a strange error where a maya library wants this new sized delete operator symbol from a Qt library instead of the libstdc++ library. And I'm guessing that this is happening because Autodesk is compiling both Qt and their own libs with the devtoolset which is somehow defining these new C++14 symbols in the headers only, while our custom build of GCC and our custom build of Qt is assuming these symbols are in the libstdc++ dynamic library. This particular error is quite strange as it only happens if I try to import a python module (our AL_USDMaya python wrappers) while being outside of maya. I can import these fine if I run this import within a normal maya session. I have tried to LD_PRELOAD our libstdc++ library but that didn't help, it seems that this libImage.so file really wants this "void operator delete[] (void* ptr, std::size_t size)" symbol to be coming from a Qt library and have the Qt_5 symbol version...

So I'm not sure what I can do now to try and configure GCC-6.3.1 to use the standard C++ library from 4.8.3... I haven't seen any gcc configure option to do this... 

We're not compiling our own gcc just for fun, it's the way we've had to configure our build toolchain as part of our rez package system in the past and it was quite convenient before on previous CentOS versions... I think we might revise this strategy as the devtoolset are now much more easily installed, and have diverged so much from a normal GCC build...

My main comment would be to further update the notes on the vfxplatform.com to reflect this newly found information as it might help others. I did notice the updates on the site with links to the source, thanks for this by the way!

If anyone has a magic clue on how to configure a gcc build to use a prebuilt libstdc++ library that would be awesome, otherwise we'll have to find a nice way of activating a devtoolset from within a rez package...

A secondary comment is about general portability. I understand why using RedHat dev toolset is alluring to vendors, but I feel this is really entrenching us deeper into RHEL/CentOS ecosystem as the toolchain looks more and more proprietary... I know it's all opensource and should be all rebuildable on other linuses, but my current experience is that it's getting really hard to reproduce the toolchain! And with RedHat now belonging to IBM I'm not quite sure that's all safe there. I think it would be nice to start looking outside of RHEL for the VFX platform in the long term and investigate other means of ensuring compatibility, maybe the ASWF collaboration might provide new opportunities?

 

Thanks and well done if you've read this whole thing!

Cheers,

Aloys

[Wayne Arnold]

you are mixing versions of Development Toolset and gcc  in the description

 

DTS 6.1 provides gcc 6.3.1

 

DTS 7.1 provides gcc 7.3.1

 

Autodesk’s build of Maya 2018 uses devtoolet-2.1  gcc 4.8.2

 

use of DTS 6.1 was for new products in CY2018

 

 

 

if you have a custom build of gcc 6.3.1 what c++ abi setting are you using ?

 

are you using:   -D_GLIBCXX_USE_CXX11_ABI=0 

 

for your compiles ?      the default for stock gcc 5.3+ has been the new ABI

 

Red Hat decided to maintain using the “old” standard for its long term stable release.

 

 

I haven’t checked on what the default will be for RHEL 8 if this has changed or not.

 

 

~ $ . /opt/rh/devtoolset-6/enable 

~ $ gcc -v

Using built-in specs.

COLLECT_GCC=gcc

COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-6/root/usr/libexec/gcc/x86_64-redhat-linux/6.3.1/lto-wrapper

Target: x86_64-redhat-linux

Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-6/root/usr --mandir=/opt/rh/devtoolset-6/root/usr/share/man --infodir=/opt/rh/devtoolset-6/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --enable-plugin --with-linker-hash-style=gnu --enable-initfini-array --disable-libgcj --with-default-libstdcxx-abi=gcc4-compatible --with-isl=/builddir/build/BUILD/gcc-6.3.1-20170216/obj-x86_64-redhat-linux/isl-install --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux

Thread model: posix

gcc version 6.3.1 20170216 (Red Hat 6.3.1-3) (GCC) 

 

 

~ $ . /opt/rh/devtoolset-7/enable 

~ $ gcc -v

Using built-in specs.

COLLECT_GCC=gcc

COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper

Target: x86_64-redhat-linux

Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-7/root/usr --mandir=/opt/rh/devtoolset-7/root/usr/share/man --infodir=/opt/rh/devtoolset-7/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --enable-plugin --with-linker-hash-style=gnu --enable-initfini-array --with-default-libstdcxx-abi=gcc4-compatible --with-isl=/builddir/build/BUILD/gcc-7.3.1-20180303/obj-x86_64-redhat-linux/isl-install --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux

Thread model: posix

gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC) 

 

--

Wayne

 

 

From: <vfx-platfo...@googlegroups.com> on behalf of Aloys Baillet <aloys....@gmail.com>

Date: Wednesday, November 7, 2018 at 6:38 AM
To: vfx-platform-discuss <vfx-platfo...@googlegroups.com>

--

You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
To post to this group, send email to vfx-platfo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/d50db208-0f45-4ebf-8bd9-8365c65f39f9%40googlegroups.com.

[olivier...@gmail.com]

Le mercredi 7 novembre 2018 06:38:31 UTC-5, Aloys Baillet a écrit :
> is assuming these symbols are in the libstdc++ dynamic library. This particular error is quite strange as it only happens if I try to import a python module (our AL_USDMaya python wrappers) while being outside of maya. I can import these fine if I run this import within a normal maya session. I have tried to LD_PRELOAD our libstdc++ library but that didn't help, it seems that this libImage.so file really wants this "void operator delete[] (void* ptr, std::size_t size)" symbol to be coming from a Qt library and have the Qt_5 symbol version...

Is your main binary python? I've had problems show up only in that case because the python binary is not linked with libstdc++. So it was getting loaded indirectly and its symbols were not globally available. I patched it with this in the python script:

ctypes.CDLL('libstdc++.so', mode=ctypes.RTLD_GLOBAL)

Not a great thing to do but mostly harmless, I think.

>
> So I'm not sure what I can do now to try and configure GCC-6.3.1 to use the standard C++ library from 4.8.3... I haven't seen any gcc configure option to do this... 

Because there isn't one. What I found is a Red Hat patch which is a large dump of assembly source and should allow building libstdc++_nonshared.a which contains the missing symbols to link with the old libstdc++.so. It is version specific.

> A secondary comment is about general portability. I understand why using RedHat dev toolset is alluring to vendors, but I feel this is really entrenching us deeper into RHEL/CentOS ecosystem as the toolchain looks more and more proprietary... I know it's all opensource and should be all rebuildable on other linuses, but my current experience is that it's getting really hard to reproduce the toolchain! And with RedHat now belonging to IBM

I agree :-) But the only alternatives I can see are:

1) Link with a newer libstdc++ and force everyone to upgrade their OS.
2) Limit the vfx platform to a much older g++.

I don't know of any good technical solution to get new C++ on old OS besides the RedHat patch. Some good documentation on how to build the toolchain on various OS might be a decent place to start. But the problem is how do you validate that a toolchain you build is actually good?

Olivier

[Aloys Baillet]

Thanks Olivier!

Unfortunately the ctypes trick doesn't work (I think LD_PRELOAD would have achieved the same effet?).

Here's the exact wording of the error: "libImage.so: symbol _ZdaPvm, version Qt_5 not defined in file libQt5Core.so.5 with link time reference" which makes me think that libQt5Core.so.5 compiled by devtoolset-6.1 (thanks Wayne for the correction!) has this nonshared static lib built in and exposed under the Qt_5 versioned symbol instead of being in the global namespace. I would say this is a build issue with Qt itself but that will be quite tricky to fix...

Cheers,

Aloys

[David Aguilar]

We ran into a similar issue and had to tweak our qt5 build to pass
"-c++std c++14" to the Qt5 ./configure script.

Can you try that out and see if it helps? It's not a quick change to
test, but figured I'd mention it in case you're running into the same
issue.

> --
> You received this message because you are subscribed to the Google Groups "vfx-platform-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vfx-platform-dis...@googlegroups.com.
> To post to this group, send email to vfx-platfo...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/vfx-platform-discuss/4464d038-3794-4c61-aaf6-da1385d2e5c3%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
David

[Aloys Baillet]

Thanks David!

I did try this already... enabling c++14 creates these symbols as undefined in Qt if compiled with our custom build of gcc... and it makes sense as these are supposed to come from libstdc++.so in our custom gcc build.

nm -D --print-file-name Qt/lib/libQt5Core.so|grep _ZdaPvm

Qt/lib/libQt5Core.so:                 U _ZdaPvm

But the Qt5Core lib compiled with devtoolset have this symbol actually defined:

nm -D --print-file-name /usr/autodesk/maya2019/lib/*.so|grep _ZdaPvm

/usr/autodesk/maya2019/lib/libQt5Core.so:000000000031f070 T _ZdaPvm

So I don't see any other way than recompiling Qt with devtoolset so the same libstdc++_nonshared.a gets stamped into the .so files the way the other maya libraries are expecting.

Cheers,

Aloys