Build tensorflow with Clang and libc++ on Linux

[Andrew Marshall]

What's the best way to go about building tensorflow wtih clang and LLVM's libc++, for Linux platforms?

Just changing CC=clang++ and `--copts=-stdlib=libc++` doesn't work due to differences in how clang (the C compiler) and clang++ (The C++ compiler) handle include paths.

Is this something best handled with a toolchain - is there an existing similar toolchain (Android must be similar?)

Thanks,

Andrew.

[Günhan Gülsoy]

Artem may have some information on this.

--
To unsubscribe from this group and stop receiving emails from it, send an email to build+un...@tensorflow.org.

[Manuel Klimek]

On Mon, Aug 10, 2020 at 9:32 PM Artem Belevich <t...@google.com> wrote:

Forwarding to Manuel (though he's on OOO ATM).

I believe we have a number of preconfigured toolchains for the build under google3/third_party/tensorflow/third_party/toolchains/preconfig/  but I'm not familiar with the details.

This may be a good place to start:

http://google3/third_party/tensorflow/third_party/toolchains/preconfig/ubuntu16.04/clang_manylinux2010-cuda10.0/cc_toolchain_config.bzl.oss?l=1125&rcl=277523803

Preconfig is all but dead (we only use it for Windows).

We now have automatic remote configuration.

 

On Mon, Aug 10, 2020 at 9:04 AM Günhan Gülsoy <gu...@google.com> wrote:

Artem may have some information on this.

On Sat, Aug 8, 2020 at 11:54 AM Andrew Marshall <planetma...@gmail.com> wrote:

What's the best way to go about building tensorflow wtih clang and LLVM's libc++, for Linux platforms?

Just changing CC=clang++ and `--copts=-stdlib=libc++` doesn't work due to differences in how clang (the C compiler) and clang++ (The C++ compiler) handle include paths.

Is this something best handled with a toolchain - is there an existing similar toolchain (Android must be similar?)

Unfortunately I don't think I have enough information to help; can you give the error log? Generally, building with clang (cuda-clang) works. Using libc++ should generally work, as we use it internally, but I haven't tried externally, and it might need a bit of fiddling to get it right.

[Andrew Marshall]

On Mon, 17 Aug 2020 at 10:29, Manuel Klimek <kli...@google.com> wrote:

Unfortunately I don't think I have enough information to help; can you give the error log? Generally, building with clang (cuda-clang) works. Using libc++ should generally work, as we use it internally, but I haven't tried externally, and it might need a bit of fiddling to get it right.

Hi Manuel - thanks for the reply.

Apologies for the lack of info - here's what I've tried so far.

Working with the current master (

1. configure with default options.

2. export CC=clang && bazel build -s --config=opt --config=libc++ --cxxopt=-stdlib=libc++ //tensorflow:tensorflow_cc

This fails on failing to find the STL include <iterator>

https://paste.ubuntu.com/p/9vFG9xFZNg/

I've tried adding the libc++ includes explicitly with `-I` or `-isystem` but get "The include path '/usr/lib/llvm-10/include/c++/v1' references a path outside of the execution root"

I could add a local repository for libc++ but it feels that that shouldn't be necessary?

Thanks for your help.

Andrew.

--

Andrew Marshall

[Manuel Klimek]

On Mon, Aug 17, 2020 at 7:32 PM Andrew Marshall <planetma...@gmail.com> wrote:

On Mon, 17 Aug 2020 at 10:29, Manuel Klimek <kli...@google.com> wrote:

Unfortunately I don't think I have enough information to help; can you give the error log? Generally, building with clang (cuda-clang) works. Using libc++ should generally work, as we use it internally, but I haven't tried externally, and it might need a bit of fiddling to get it right.

Hi Manuel - thanks for the reply.

Apologies for the lack of info - here's what I've tried so far.

Working with the current master (

1. configure with default options.

2. export CC=clang && bazel build -s --config=opt --config=libc++ --cxxopt=-stdlib=libc++ //tensorflow:tensorflow_cc

This fails on failing to find the STL include <iterator>

https://paste.ubuntu.com/p/9vFG9xFZNg/

Does /usr/bin/clang -c test.cc -std=c++14 -stdlib=libc++ works, if test.cc contains #include <iterator>?

I've tried adding the libc++ includes explicitly with `-I` or `-isystem` but get "The include path '/usr/lib/llvm-10/include/c++/v1' references a path outside of the execution root"

Yea, that is the interesting part -- if clang -x c++ does not output the right paths here, they'll not be correctly included.

Can you paste the output of

/usr/bin/clang -E -x c++ - -v

[Andrew Marshall]

On Mon, 17 Aug 2020 at 18:59, Manuel Klimek <kli...@google.com> wrote:

Does /usr/bin/clang -c test.cc -std=c++14 -stdlib=libc++ works, if test.cc contains #include <iterator>?

Yes this is fine.

 

I've tried adding the libc++ includes explicitly with `-I` or `-isystem` but get "The include path '/usr/lib/llvm-10/include/c++/v1' references a path outside of the execution root"

Yea, that is the interesting part -- if clang -x c++ does not output the right paths here, they'll not be correctly included.

Can you paste the output of

/usr/bin/clang -E -x c++ - -v

Output pasted here: https://paste.ubuntu.com/p/5Vywmb8wX2/

The include path for libc++ is not present, instead it has the include paths for GNU libstdc++.

Running 

`/usr/bin/clang -E -x c++ -stdlib=libc++ - -v`

outputs the following:

https://paste.ubuntu.com/p/9HSfsTkw8g/

Which has the correct paths.

--

Andrew Marshall

[Manuel Klimek]

Are you configuring with cuda or cpu only? 

[Andrew Marshall]

On Wed, 19 Aug 2020 at 11:15, Manuel Klimek <kli...@google.com> wrote:

Are you configuring with cuda or cpu only? 

Sorry - ignore that previous reply.

CPU only build (just accepting the linux defaults). My `.tf_configure.bazelrc` contains

```

build --action_env PYTHON_BIN_PATH="/home/andrew/.pyenv/versions/3.8.3/bin/python3"
build --action_env PYTHON_LIB_PATH="/home/andrew/.pyenv/versions/3.8.3/lib/python3.8/site-packages"
build --python_path="/home/andrew/.pyenv/versions/3.8.3/bin/python3"
build --config=xla
build:opt --copt=-march=native
build:opt --copt=-Wno-sign-compare
build:opt --host_copt=-march=native
build:opt --define with_default_optimizations=true
test --flaky_test_attempts=3
test --test_size_filters=small,medium
test:v1 --test_tag_filters=-benchmark-test,-no_oss,-gpu,-oss_serial
test:v1 --build_tag_filters=-benchmark-test,-no_oss,-gpu
test:v2 --test_tag_filters=-benchmark-test,-no_oss,-gpu,-oss_serial,-v1only
test:v2 --build_tag_filters=-benchmark-test,-no_oss,-gpu,-v1only
build --action_env TF_CONFIGURE_IOS="0"

```

--

Andrew Marshall

[Manuel Klimek]

Ok, for the cuda toolchain I could tell you how to generate a patch, for the cc_configure toolchain looping in +Lukacs Berki to find the right person to explain how to configure libc++ in a way that figures out the right builtin-include-paths.

[Andrew Marshall]

On Wed, 19 Aug 2020 at 12:09, Manuel Klimek <kli...@google.com> wrote:

Ok, for the cuda toolchain I could tell you how to generate a patch, for the cc_configure toolchain looping in +Lukacs Berki to find the right person to explain how to configure libc++ in a way that figures out the right builtin-include-paths.

Thanks for your help. If you could point me in the right direction for the cuda toolchain that would be helpful, since it's likely I'll be trying that shortly anyway.

Cheers,

Andrew. 

--

Andrew Marshall

[Manuel Klimek]

This line:

https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl#L277

needs to add -stdlib=libc++, you can just patch it to try it out, or look at what I did for sysroot if you want a full patch (we're using sysroot for our manylinux cross-compile)

[Andrew Marshall]

Thanks for the tip, 

I installed the CUDA dependencies but fell at pretty much the first hurdle. Working with an unmodified v2.3.0 checkout,

```

 bazel build --config=opt --config=cuda_clang  //tensorflow:tensorflow_cc

```

fails with 

```

ERROR: /home/andrew/.cache/bazel/_bazel_andrew/05f0c1fc0fd5d93d6530b840ab375f68/external/zlib/BUILD.bazel:5:1: undeclared inclusion(s) in rule '@zlib//:zlib':
this rule is missing dependency declarations for the following files included by 'external/zlib/compress.c':
  '/usr/lib/clang/10.0.0/include/stddef.h'
  '/usr/lib/clang/10.0.0/include/__stddef_max_align_t.h'
  '/usr/lib/clang/10.0.0/include/limits.h'
  '/usr/lib/clang/10.0.0/include/stdarg.h

```

Note this is nothing to do with building libc++, I haven't got that far yet...

Will investigate myself but any pointers appreciated.

Thanks,

Andrew.

--

Andrew Marshall

[Andrew Marshall]

I don't know if it's connected, but when I call

```

clang -E -xc - -v

```

It lists among the system headers

```

/usr/lib/llvm-10/lib/clang/10.0.0/include

```

and not

```

/usr/lib/clang/10.0.0/include

```

Although both are present on my system. I installed clang-10 using the apt packages provided by LLVM.

--

Andrew Marshall

[Andrew Marshall]

OK, patching `cuda_configure.bzl` with the additional clang include path appears to have done the trick. I'll see if I can work out why the include path is different and submit a proper patch.

--

Andrew Marshall

[Andrew Marshall]

On Fri, 18 Sep 2020 at 18:40, Artem Belevich <t...@google.com> wrote:

Having multiple instances of clang include headers suggests that it has not been installed correctly or there's something wrong with the set of packages you have installed.

Clang's include path is somewhat special, and specifying it explicitly when clang didn't do it itself will put it in the wrong place in the search list and that will likely lead to more problems, some obvious, some not.

I'd recommend figuring out which packages have installed those two copies of clang includes and figure out which one of them should be gone.

`dpkg -S /usr/lib/clang/10.0.0` and `dpkg -S /usr/lib/llvm-10` should give you the names of the packages.

You need the one from the package set that installed the clang binary you are using. Normally, I'd expect clang to be able to find its own headers, but it, apparently, didn't happen in your case.

--Artem

Hi Artem, thanks for the reply,

There aren't multiple instances, /usr/lib/clang/10.0.0/include is a symlink to /usr/lib/llvm-10/lib/clang/10.0.0/include. dpkg reports it was put there by libclang-common-10-dev, which is a dependency of the clang-10 package.

The other problem I'm facing is that the `-no-canonical-prefixes` (which seems to be added by just about everything) and `-stdlib=libc++` flags don't play well together.

I'll maybe try a docker image with a clean environment. 

Thanks again for your help.

 

--

Andrew Marshall

[Andrew Marshall]

On Fri, 18 Sep 2020 at 22:21, Artem Belevich <t...@google.com> wrote:

I'll maybe try a docker image with a clean environment. 

That would definitely have a better chance to work out of the box. That's what I usually use when I need to tinker with TF.

--Artem

So I fired up a fresh Docker container and built and installed Clang 10 from source rather than using the apt packages. Tensorflow v2.3.0 compiled with no problems, with libc++, out of the box.

It indeed looks like the symlinked path was the cause of all those problems. I'll put something up on GitHub issues so that anyone else with the same problem can find it.

Thanks for your help.

--

Andrew Marshall

[Vartika Singh]

++ @Michael O'Connor (Deep Learning) @Cliff Woolley @Nathan Luehr

 

From: Andrew Marshall <planetma...@gmail.com>
Date: Friday, September 18, 2020 at 10:19 AM
To: Manuel Klimek <kli...@google.com>
Cc: Lukacs Berki <lbe...@google.com>, Artem Belevich <t...@google.com>, Günhan Gülsoy <gu...@google.com>, SIG Build <bu...@tensorflow.org>
Subject: Re: Build tensorflow with Clang and libc++ on Linux

 

External email: Use caution opening links or attachments

 

--

[Vartika Singh]

+ @Masoud Charkhabi

[Alexander Grund]

We had this issue as well and I opened a report for that: https://github.com/tensorflow/tensorflow/issues/33975

A decent fix is https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0_fix-cuda-build.patch which should work for your case as well.

Am 17.09.21 um 19:36 schrieb 'Vartika Singh' via SIG Build:

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dipl.-Inf. Alexander Grund
Research Assistant

Technische Universität Dresden
Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH)
01062 Dresden
Tel.: +49 (351) 463-35982
E-Mail: alexand...@tu-dresden.de
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Manuel Klimek]

On Mon, Sep 20, 2021 at 9:15 AM Alexander Grund <alexand...@tu-dresden.de> wrote:

We had this issue as well and I opened a report for that: https://github.com/tensorflow/tensorflow/issues/33975

A decent fix is https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.1.0_fix-cuda-build.patch which should work for your case as well.

It would be good if we could understand this problem better, so we don't need workarounds. Do you by chance have a Dockerfile that in the end reproduces the problem?

We do pass -no-canonical-* to gcc in order to make it not resolve symlinks.