About differences between upstream clang and aosp prebuilt clang for building the aosp

[Minseong Kim]

Hi~ 

We know that the upstream clang can also be used to compile the aosp master (of course some minor patches for disabling -Werror).

The aosp prebuilt clang seems to be compiled with some llvm/clang sub-projects such as clang-tools-extra, compiler-rt, libcxx, libcxxabi, libunwind, and openmp.

I have a couple of basic questions regarding this as the following:

1. If the upstream clang is built with the same set of sub-projects as the aosp prebuilt clang, and use it to build the aosp, what is the implication imposed on the resulting binaries ? In other words, what are the differences on the resulting binaries (in terms of both performance and functionality) ?

2. What is the purpose having two source directories of libunwind and libunwind_llvm ?

3. Should the external project jemalloc also be compiled together with the upstream clang to match for the aosp prebuilt ?

Thank you in advance for your kindness.

cheers,

-- Min

[Renato Golin]

On 24 August 2016 at 13:03, Minseong Kim <minseo...@linaro.org> wrote:
> The aosp prebuilt clang seems to be compiled with some llvm/clang
> sub-projects such as clang-tools-extra, compiler-rt, libcxx, libcxxabi,
> libunwind, and openmp.

Hi Min,

For building code, clang-tools-extra is useless, as it's home for
Clang side projects only.

> 1. If the upstream clang is built with the same set of sub-projects as the
> aosp prebuilt clang, and use it to build the aosp, what is the implication
> imposed on the resulting binaries ? In other words, what are the differences
> on the resulting binaries (in terms of both performance and functionality) ?

If you don't use those libraries directly (via -lc++ -lomp etc), than
it makes no difference.

I'm assuming you're building your upstream Clang without any component
other than clang and llvm.

In that case, it's using the libraries on your sysroot, which could be
GNU's libstdc++, libgcc, etc.

So, a quick run down:

clang-tool-extra: irrelevant
compiler-rt: replacement of some parts of libgcc
libcxx/abi: replacement for libstdc++/libgcc_eh
libunwind: replacement for unwind libraries (including the "other"
libunwind and libgcc_s)
openmp: irrelevant, as I don't think Android uses it at all (does it?)

All of them need compiler flags and are not enabled by default. So, if
you were using those options and not building the packages, it would
be breaking the build. Since you can build with just Clang, I'm
assuming you're not using any of them, therefore, they're all are
irrelevant.

I remember hearing that Android was trying to use libc++ and
compiler-rt, but I don't know the status of it. If it does, it's
probably enabled through some build flags, in which case, you need to
build it in the way Android expects to find it.

In the case of compiler-rt, it's *not* how it's built upstream. So, I
advise against using it now, and let Google fix it first.

However, it should be fine to use libc++ and libunwind with the
relevant options.

> 2. What is the purpose having two source directories of libunwind and
> libunwind_llvm ?

They're probably *different* libunwinds. For a very unfortunate
decision, LLVM's unwind library is called "libunwind", but there was
already another unwind library with the same name [1].


cheers,
--renato


[1] http://www.nongnu.org/libunwind/

[Stephen Hines]

On Wed, Aug 24, 2016 at 5:26 AM, Renato Golin <renato...@linaro.org> wrote:

On 24 August 2016 at 13:03, Minseong Kim <minseo...@linaro.org> wrote:
> The aosp prebuilt clang seems to be compiled with some llvm/clang
> sub-projects such as clang-tools-extra, compiler-rt, libcxx, libcxxabi,
> libunwind, and openmp.

Hi Min,

For building code, clang-tools-extra is useless, as it's home for
Clang side projects only.



> 1. If the upstream clang is built with the same set of sub-projects as the
> aosp prebuilt clang, and use it to build the aosp, what is the implication
> imposed on the resulting binaries ? In other words, what are the differences
> on the resulting binaries (in terms of both performance and functionality) ?

The difference is that we don't support using upstream Clang today. Our only supported toolchain is the one that we build from Android's sources (because we do use cherry-picks, and thus it is harder for you to get an exact match on versioning). I expect this to get better in the near future, but we aren't there yet.

 

If you don't use those libraries directly (via -lc++ -lomp etc), than
it makes no difference.

I'm assuming you're building your upstream Clang without any component
other than clang and llvm.

In that case, it's using the libraries on your sysroot, which could be
GNU's libstdc++, libgcc, etc.

So, a quick run down:

clang-tool-extra: irrelevant

We actually do build and use clang-tidy now.

 

compiler-rt: replacement of some parts of libgcc

Android has been using compiler-rt on the device (as part of RenderScript), and it also is used to provide some missing functionality for things like ftrapv (the overflow functions plain don't exist in libgcc, so we need to pull them from somewhere). We aren't officially switched over to compiler-rt in place of libgcc, but that will be happening soon.

 

libcxx/abi: replacement for libstdc++/libgcc_eh

libc++/abi is indeed the STL that the Android platform uses. It is also the future preferred STL for the NDK, although again it isn't 100% yet. Dan Albert has been making great strides there, and I am looking forward to being able to recommend libc++ for all Android developers.

 

libunwind: replacement for unwind libraries (including the "other"
libunwind and libgcc_s)

Yes, we do use libunwind_llvm and also libunwind. Android needs both because neither fully supports all of our relevant device targets. :(

 

openmp: irrelevant, as I don't think Android uses it at all (does it?)

Not true. We added openmp recently, because it was added to gcc in the NDK at some point. To maintain parity there, we figured it was best to add LLVM's version. Note that openmp is not something that the Android platform uses at all, so you have to bundle everything for it in any app.

 

All of them need compiler flags and are not enabled by default. So, if
you were using those options and not building the packages, it would
be breaking the build. Since you can build with just Clang, I'm
assuming you're not using any of them, therefore, they're all are
irrelevant.

I remember hearing that Android was trying to use libc++ and
compiler-rt, but I don't know the status of it. If it does, it's
probably enabled through some build flags, in which case, you need to
build it in the way Android expects to find it.

We already use libc++ for the platform (and have done so since at least Lollipop). libc++ just isn't the default for the NDK yet, because of the need for some support library help. We also do use compiler-rt as I mentioned above. It is also important to have compiler-rt because we do rely on the sanitizers, which are all built out of compiler-rt too.

 

In the case of compiler-rt, it's *not* how it's built upstream. So, I
advise against using it now, and let Google fix it first.

This is something that I was working on, but it has been set aside for more pressing issues. I don't expect that the next Android release after Nougat will use the upstream build system either, so I would suggest that you try to adapt more closely to our existing builds if you want to experiment. We post all of our instructions for producing the exact same builds we ship, and it is very easy to interact with upstream for cherry-picks/merges. The only hindrance is the need to modify Android.mk files occasionally. Considering the troubles I have seen with Cmake over the past few months, I am not sure which is honestly better/worse, since the Cmake-based version makes it nearly impossible to create multiple working cross-compilers.

 

However, it should be fine to use libc++ and libunwind with the
relevant options.


> 2. What is the purpose having two source directories of libunwind and
> libunwind_llvm ?

They're probably *different* libunwinds. For a very unfortunate
decision, LLVM's unwind library is called "libunwind", but there was
already another unwind library with the same name [1].

Precisely. We do use the other libunwind for some device targets, so we can't have the same name.

Thanks,

Steve

 

cheers,
--renato


[1] http://www.nongnu.org/libunwind/

--
You received this message because you are subscribed to the Google Groups "android-llvm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-llvm+unsubscribe@googlegroups.com.
To post to this group, send email to androi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Renato Golin]

On 24 August 2016 at 18:12, Stephen Hines <srh...@google.com> wrote:
> The difference is that we don't support using upstream Clang today. Our only
> supported toolchain is the one that we build from Android's sources (because
> we do use cherry-picks, and thus it is harder for you to get an exact match
> on versioning). I expect this to get better in the near future, but we
> aren't there yet.

Well, that's part of Min's job, too.

I think the hard question here was: what's the *actual* difference in
conformance and performance, assuming the toolchains have the same
components.

The answer is, unfortunately, "it depends". Overall, you should have
very little changes in terms of hardware conformance and performance,
since the in-tree build is getting closer to trunk, I believe. But you
probably will see packages fail due to the lack of cherry-picks and
internal patches (if any).

It would be good to know the differences, though, so we can focus on
the upstreaming the remaining problems (independently of the Android
release schedule).

> We actually do build and use clang-tidy now.

To build packages?

> Yes, we do use libunwind_llvm and also libunwind. Android needs both because
> neither fully supports all of our relevant device targets. :(

That's... unfortunate... Maybe we could help implement the missing features?

> We already use libc++ for the platform (and have done so since at least
> Lollipop). libc++ just isn't the default for the NDK yet, because of the
> need for some support library help. We also do use compiler-rt as I
> mentioned above. It is also important to have compiler-rt because we do rely
> on the sanitizers, which are all built out of compiler-rt too.

Still, the way Android builds LLVM is different (for now), so we may
have trouble putting libraries in the right places. But it would be
good to know the differences, so we should get a list of problems, if
any, and work towards them.

> Considering the troubles I have seen with Cmake over the past few months, I
> am not sure which is honestly better/worse, since the Cmake-based version
> makes it nearly impossible to create multiple working cross-compilers.

I stopped following the CMake thread, was there any more work towards
cross compiling RT and libc++?

cheers,
--renato

[Stephen Hines]

On Wed, Aug 24, 2016 at 1:41 PM, Renato Golin <renato...@linaro.org> wrote:

On 24 August 2016 at 18:12, Stephen Hines <srh...@google.com> wrote:
> The difference is that we don't support using upstream Clang today. Our only
> supported toolchain is the one that we build from Android's sources (because
> we do use cherry-picks, and thus it is harder for you to get an exact match
> on versioning). I expect this to get better in the near future, but we
> aren't there yet.

Well, that's part of Min's job, too.

I think the hard question here was: what's the *actual* difference in
conformance and performance, assuming the toolchains have the same
components.

That is what I was trying to answer. We won't officially support any other toolchains than the one we build and qualify. Assuming you are happy with validating your own toolchain, that's great. But coming to us with bugs that don't reproduce for our toolchain will probably not be met with much help. I would really like to get to the point where we can use the upstream build/sources precisely, but I can't tell when that is going to happen.

 

The answer is, unfortunately, "it depends". Overall, you should have
very little changes in terms of hardware conformance and performance,
since the in-tree build is getting closer to trunk, I believe. But you
probably will see packages fail due to the lack of cherry-picks and
internal patches (if any).

The next (current) update should bring us to a point where there are no internal patches to LLVM. The only differences will be in cherry-picks that have been applied ahead of the regular schedule. As we get better at this, the number of cherry-picks should diminish too, because we can do faster validation, and thus release more frequently (i.e. fewer time gaps where Android-related things are broken in upstream).

 

It would be good to know the differences, though, so we can focus on
the upstreaming the remaining problems (independently of the Android
release schedule).

Everything is submitted upstream actually. That's all folks. Thanks to Pirama for doing the actual hard work here, and Richard Smith for reviewing/shepherding many of those patches. As I said, our next update should result in an equivalent snapshot of something upstream, as we back out all the remaining local hacks in favor of the proper upstream equivalent patches.

 

> We actually do build and use clang-tidy now.

To build packages?

Not to build, but we do have buildbot configurations that are running with clang-tidy enabled.

 

> Yes, we do use libunwind_llvm and also libunwind. Android needs both because
> neither fully supports all of our relevant device targets. :(

That's... unfortunate... Maybe we could help implement the missing features?

There are some plans forming about what to do here, but nothing I can share yet unfortunately.

 

> We already use libc++ for the platform (and have done so since at least
> Lollipop). libc++ just isn't the default for the NDK yet, because of the
> need for some support library help. We also do use compiler-rt as I
> mentioned above. It is also important to have compiler-rt because we do rely
> on the sanitizers, which are all built out of compiler-rt too.

Still, the way Android builds LLVM is different (for now), so we may
have trouble putting libraries in the right places. But it would be
good to know the differences, so we should get a list of problems, if
any, and work towards them.


> Considering the troubles I have seen with Cmake over the past few months, I
> am not sure which is honestly better/worse, since the Cmake-based version
> makes it nearly impossible to create multiple working cross-compilers.

I stopped following the CMake thread, was there any more work towards
cross compiling RT and libc++?

I think there were still some folks working on it, and it might have made substantial progress since I stopped looking at it too. Unfortunately, building from upstream was not considered the highest priority item by Android management, so other things have been occupying my time more recently.

Thanks,

Steve

 

cheers,
--renato

[Renato Golin]

On 25 August 2016 at 03:21, Stephen Hines <srh...@google.com> wrote:
> That is what I was trying to answer. We won't officially support any other
> toolchains than the one we build and qualify. Assuming you are happy with
> validating your own toolchain, that's great. But coming to us with bugs that
> don't reproduce for our toolchain will probably not be met with much help.

I think we have crossed wires...

Linaro has an initiative to make Android compile with LLVM, which,
IIRC, we have identified in the past that the best way forward (for
now) would be to have:

* Vanilla LLVM at some point in trunk + cherry-picks from trunk,
without internal patches
* Built using LLVM's own CMake files, for *all* projects (including
RT, libc++, etc).

Of course we're not there yet, but our goal is to get there, and Min
is trying to probe the next steps in that journey, which is invariably
affected by Google's roadmap.

We most definitely will *not* create a new toolchain.

But I want to avoid the problems in the past where Android's LLVM's
patches could not be upstreamed for the same reason.

Building it with Vanilla LLVM seems like a good way to understanding
the problem, and it seems Min had some degree of success, so I don't
see why would this be a bad move.

More importantly, we're *not* asking for "official support" of any
kind, we actually want to *provide* support, to get a more upstream
version of LLVM being used, at least for AOSP.

I don't think it makes any sense having an open source project that
can only be compiled by a toolchain that is controlled by any specific
corporation.

The outcome is this thread, which is confusing to read and understand,
hard to offer help and collaborate, and we're still not able to
compile with a vanilla open source toolchain.

> The next (current) update should bring us to a point where there are no
> internal patches to LLVM. The only differences will be in cherry-picks that
> have been applied ahead of the regular schedule.

Just to be clear, these cherry-picks are from far ahead in LLVM's
trunk, not cherry-pick from an old Android-LLVM tree, right?

> Everything is submitted upstream actually. That's all folks. Thanks to
> Pirama for doing the actual hard work here, and Richard Smith for
> reviewing/shepherding many of those patches.

I'm still unclear as to what you mean by "everything". Is Compiler-RT
fixed? Libc++? Libunwind?

If not, than that's not *everything*.

Or is it "everything that is important for AOSP, but not Android"? In
this case, that's good enough for me.

> Not to build, but we do have buildbot configurations that are running with
> clang-tidy enabled.

Interesting, so you get your current source repo, pass clang-tidy on
them, then compile? How can you control where the bugs come from, if
they appear?

This sounds like a lot of work. It would be easier to pass it once and
push the patch upstream so that everyone profits.

cheers,
--renato

[Stephen Hines]

On Thu, Aug 25, 2016 at 3:15 AM, Renato Golin <renato...@linaro.org> wrote:

On 25 August 2016 at 03:21, Stephen Hines <srh...@google.com> wrote:
> That is what I was trying to answer. We won't officially support any other
> toolchains than the one we build and qualify. Assuming you are happy with
> validating your own toolchain, that's great. But coming to us with bugs that
> don't reproduce for our toolchain will probably not be met with much help.

I think we have crossed wires...

Yes, I think so too. Every few months, one of these emails is sent to me, which leaves me confused as to what Linaro actually intends to do.

 

Linaro has an initiative to make Android compile with LLVM, which,
IIRC, we have identified in the past that the best way forward (for
now) would be to have:

* Vanilla LLVM at some point in trunk + cherry-picks from trunk,
without internal patches

Agreed.

 

* Built using LLVM's own CMake files, for *all* projects (including
RT, libc++, etc).

Agreed, but this is the part that requires more work upstream, which we haven't prioritized right now. If this is something that Linaro can fix, that would be great.

 

Of course we're not there yet, but our goal is to get there, and Min
is trying to probe the next steps in that journey, which is invariably
affected by Google's roadmap.

We most definitely will *not* create a new toolchain.

It did sound like you wanted to qualify upstream builds (or build numbers) for use in compiling aosp/master. That is the part that I want to make sure comes with giant red warning flags right now. We (Google Android) aren't ready for that just yet, but we will get there.

 

But I want to avoid the problems in the past where Android's LLVM's
patches could not be upstreamed for the same reason.

Building it with Vanilla LLVM seems like a good way to understanding
the problem, and it seems Min had some degree of success, so I don't
see why would this be a bad move.

More importantly, we're *not* asking for "official support" of any
kind, we actually want to *provide* support, to get a more upstream
version of LLVM being used, at least for AOSP.

I don't think it makes any sense having an open source project that
can only be compiled by a toolchain that is controlled by any specific
corporation.

We aren't doing that, but we also aren't facilitating the use of other toolchains either. For instance, we are removing GCC-specific workarounds in Android-owned code, because we are turning down all of our GCC builds now that everything can be built with clang (modulo Valgrind, which is being fixed). As far as "can only be compiled", I think that this is an extremely harsh phrasing. We aren't forcing anyone to use Clang. It is just the only configuration that we are officially "supporting". There is a difference between telling everyone that they "can't" use their favorite compiler, and us saying that we don't test or prioritize fixes for other unsupported compilers. Everyone is free to use whatever they like, but I want to be realistic and open about what we are/aren't doing.

 

The outcome is this thread, which is confusing to read and understand,
hard to offer help and collaborate, and we're still not able to
compile with a vanilla open source toolchain.

On the contrary, I actually think that a vanilla toolchain has been able to compile Android for a long time. I just hesitate to recommend that path, as we haven't had the time to validate that configuration. If I say it is officially supported, I will start seeing a flood of bugs whenever something breaks. I would rather be able to set up infrastructure to catch these cases before officially declaring success.

 

> The next (current) update should bring us to a point where there are no
> internal patches to LLVM. The only differences will be in cherry-picks that
> have been applied ahead of the regular schedule.

Just to be clear, these cherry-picks are from far ahead in LLVM's
trunk, not cherry-pick from an old Android-LLVM tree, right?

These are cherry-picks that are 100s of CLs into the future of upstream LLVM. We were on r271374, and are currently testing r275480. I suppose there is one additional patch to LLVM that isn't present in upstream, but it is for a Windows-specific problem (basically related to SHGetKnownFolderPath not being available in our mingw). This only affects the Windows NDK build, so I don't consider it a problem for testing/building the actual platform (which doesn't use the Windows builds at all).

 

> Everything is submitted upstream actually. That's all folks. Thanks to
> Pirama for doing the actual hard work here, and Richard Smith for
> reviewing/shepherding many of those patches.

I'm still unclear as to what you mean by "everything". Is Compiler-RT
fixed? Libc++? Libunwind?

Android had no divergence for libc++ or libunwind, and every other LLVM-related project has been patched upstream (minus the one windows NDK issue, which doesn't affect building Android devices).

 

If not, than that's not *everything*.

Or is it "everything that is important for AOSP, but not Android"? In
this case, that's good enough for me.


> Not to build, but we do have buildbot configurations that are running with
> clang-tidy enabled.

Interesting, so you get your current source repo, pass clang-tidy on
them, then compile? How can you control where the bugs come from, if
they appear?

I mean that Android has an internal aosp/master buildbot that builds everything using clang-tidy (in addition to clang). It only adds (a ton of) warnings to the build, so there are engineers trying to fix a good chunk of those issues too (in addition to other warning cleanup work that has been ongoing for aosp/master). We most certainly aren't at 0 warnings yet for either regular builds or clang-tidy, but that is eventually going to be a goal for Android.

Thanks,

Steve

[Renato Golin]

On 25 August 2016 at 19:27, Stephen Hines <srh...@google.com> wrote:
> Agreed, but this is the part that requires more work upstream, which we
> haven't prioritized right now. If this is something that Linaro can fix,
> that would be great.

Last time, we "agreed" Google would look into that and we'd look for
other issues. You sent me the build instructions from your progress.

If this is the last big thing, and you can't prioritise it right now,
then we should definitely have a look.

> It did sound like you wanted to qualify upstream builds (or build numbers)
> for use in compiling aosp/master. That is the part that I want to make sure
> comes with giant red warning flags right now. We (Google Android) aren't
> ready for that just yet, but we will get there.

Absolutely.

We (Linaro) deal upstream, so everything is "as is", no promises. I
definitely don't expect Google to promise anything, nor I want to
encourage people to report bugs in upstream LLVM to Google.

Actually, I'd be very happy if people started reporting the bugs
upstream (LLVM bugzilla), but for that, as you said, it needs to not
only be an upstream version, but built using upstream CMake files,
like everyone else.

> We aren't doing that, but we also aren't facilitating the use of other
> toolchains either. For instance, we are removing GCC-specific workarounds in
> Android-owned code, because we are turning down all of our GCC builds now
> that everything can be built with clang (modulo Valgrind, which is being
> fixed).

Right, I don't expect Google to prioritise everything at the same
time, and we agreed earlier that this is a multiple step process. By
no means I'm trying to force you to prioritise anything, I'm just
trying to figure out what you are not prioritising, so Linaro can work
on it. No promises, no support needed.

> As far as "can only be compiled", I think that this is an extremely
> harsh phrasing.

Apologies, I didn't intend it that way. I was merely expressing my
long term goals for any open source projects.

My point is that AOSP should be like *any* other open source product,
including the Linux kernel, all Linux distributions, FreeBSD kernel
and userland, etc. in that it should "just build" with an upstream
compiler with minimal modifications.

This is not about Android, nor about Google's priorities or support
contracts. This is just about AOSP and LLVM.

I constantly receive bug reports about Android, OpenEmbedded, FreeBSD
and Linux distributions that are hard to demonstrate upstream. Of all
groups, Android is the hardest to reproduce because of how LLVM is
built. So, this is not just a personal opinion or crusade, this is
about doing my job correctly.

With all the others, I can always get to the bottom of things. With
Android, most of the time, I reach dead ends, because it is impossible
to reproduce a problem, that could very well be in LLVM, but doesn't
happen upstream at all.

I know you are doing your best, and I also appreciate how priorities
change, and that's precisely why I'm asking for your input, so we can
help you achieve that goal, at least for AOSP.

> On the contrary, I actually think that a vanilla toolchain has been able to
> compile Android for a long time.

I think we use different meanings of the word "vanilla". Can AOSP be
entirely compiled (except the kernel) by the binaries from
http://llvm.org/releases/?

> I just hesitate to recommend that path, as
> we haven't had the time to validate that configuration. If I say it is
> officially supported, I will start seeing a flood of bugs whenever something
> breaks. I would rather be able to set up infrastructure to catch these cases
> before officially declaring success.

This is probably your misunderstanding. I have said before, but I want
to make it clear again.

Linaro will not ask for Google's support in any way, shape or form.
We're here to help. If you can tell us what's missing from the vanilla
LLVM build, even if that's validation, then we can maybe work on it.

If I don't know what's missing, I can't help you.

You say AOSP builds with vanilla LLVM, but I know RT can't cross
build, and we're missing some additions to RT (128-bit rounding, for
instance), so that's not working for ARM and AArch64 upstream.

You say LLVM's libunwind can't completely replace the old one, but I
don't know what's missing. Unless bugs are filled, there's no way for
us (LLVM developers) to know what to do to complete AOSP support.

I'm guessing from your response that Clang+LLVM+libcxx is ok, so Min
can start there.

> I mean that Android has an internal aosp/master buildbot that builds
> everything using clang-tidy (in addition to clang). It only adds (a ton of)
> warnings to the build, so there are engineers trying to fix a good chunk of
> those issues too (in addition to other warning cleanup work that has been
> ongoing for aosp/master). We most certainly aren't at 0 warnings yet for
> either regular builds or clang-tidy, but that is eventually going to be a
> goal for Android.

Ah, excellent! Ok, good to know these tools are helping. :)

Still, for Min's work, it's irrelevant. I use the term in the loose
sense, as in "he shouldn't worry about it", not as in "no one cares
about it".

Hope that makes sense.

cheers,
--renato

[Minseong Kim]

Hi Stephen and Renato,

Thank you guys for providing a lot of relevant information regarding my questions.

I see the discussion has been heated up a bit :( I asked the questions to take the shortest path as to otherwise, one has to look at the code to see the differences (which can be extremely time-consuming, of course).

Our (actually your) discussion raised many more follow-up works but I can see the difficulties of getting our goals and priorities precisely synced up for inevitable reasons.

I will think this discussion over again so that we all could be benefiting.

cheers,

-- Min