Generate external/llvm Intrinsics.gen and Attributes.inc on android10

[Paschalis Mpeis]

Hi there,

I am building some tools from external/llvm, specifically opt, llvm-link, and llc.

For this, I revived the relevant Android.mk from previous commits

(for each of the tools, and a relevant portion at external/llvm/Android.mk).

They are normally picked up by the soong build system, and I can for example call: `m opt`.

However, these tools depend on two intermediates:

• target/product/<DEVICE>/gen/EXECUTABLES/opt_intermediates/llvm/IR/Attributes.inc
  
• target/product/<DEVICE>/gen/EXECUTABLES/opt_intermediates/llvm/IR/Intrinsics.gen

I have enabled FORCE_BUILD_LLVM_COMPONENTS (see here)

and I can see there are relevant entries for what I want at lines: 148 and  154.

So, is my question is: is there a way to generate those 2 intermediate files using the existing Android.bp setup on android10-release branch?

Cheers,

Paschalis

[Colin Cross]

Those files are generated by the llvm_tblgen modules in external/llvm/Android.bp.  You'll find their outputs at:

out/soong/.intermediates/external/llvm/llvm-gen-attributes/gen/llvm/IR/Attributes.inc

out/soong/.intermediates/external/llvm/llvm-gen-intrinsics/gen/llvm/IR/Intrinsics.gen

As a hack you could copy those files to the places where the Android.mk files create them, just be aware they will not be updated if you make changes to the source files and rebuild.  You could also try reverting the change that removed generating those files in Make:

commit 7ebb1e3516ebd630aa3fe94f9b4cca2c2dbb09ae
Author: Dan Willemsen <dwill...@google.com>
Date:   Thu Nov 1 00:16:34 2018 -0700

    Disable tblgen in Make
   
    The last user of these have been removed. simpleperf is still using
    LLVM_HOST_BUILD_MK and LLVM_DEVICE_BUILD_MK, so these can't all be
    removed yet.
   
    Test: treehugger
    Change-Id: I0b3966fa5630e9795a1e7d7b9658f9a2bcad5271

You'll likely have better success building opt, llvm-link and llc using Android.bp files than trying to get the old Android.mk files to work though.

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to
android-buildi...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-buildi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/6e937488-80cb-42c8-b15c-087cafb7c400%40googlegroups.com.

[Paschalis Mpeis]

Thanks a lot Colin for taking the time to read through my message and reply.

I ended up using Android.bp files as you suggested.

Since all these tools were dropped from building in android10, I though they were never converted to blueprint files.

It turns out commit 7623033 had them, and with minor changes I made them compiling for target device.

android-...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-...@googlegroups.com.

[Paschalis Mpeis]

Hi again,

I hope everybody is doing well given these days' circumstances!

I have a follow-up question on my previous attempts to use LLVM.

What I want is to include libLLVM_android on libart-compiler and use it on an arm64 device (sailfish) in particular.

Now, I can successfully compile llvm components I need using:

m libLLVM_android opt llc llvm-linker

Then I install the library and the 3 tools on a sailfish device, and when executing llc for example I can see that it can successfully utilize the libLLVM_android library.

But when I try to include the libLLVM_android on art/compiler/Android.bp I get issues.

Here's my changes:

art/compiler/Android.bp:

libart_cc_defaults {
    name: "some_llvm_defaults",
    defaults: [
        // I am not using llvm-defaults as compilation will break on warnings like: -Wc++14-extensions
        // "llvm-defaults",
        "force_build_llvm_components",
    ],
    target: {
        android: {
            srcs: [
                // my additional .cc files go here..
            ],
        },
    },
    // I think this is the proper way to include the headers
    header_libs: ["llvm-headers"],
    // not sure about the next 2 lines
    // export_shared_lib_headers: ["libLLVM_android"],
    // compile_multilib: "first",

    // that's how I should include the library I think, but it gives me the error#1 listed below
    shared_libs: ["libLLVM_android"],
    cflags: [
        // some flags necessary for LLVM
        "-D__STDC_LIMIT_MACROS",
        "-D__STDC_CONSTANT_MACROS",
        "-Wno-unused-parameter",
    ],
    // headers shouldn't be included like this I think, but instead
    // with someting like header_libs that I did above
    include_dirs: [
        "external/llvm/include",
    ],
}

On the external/llvm/Android.bp I can see here that it does export the relevant header directories for android targets,

and since I have successfully compiled and installed libLLVM_android on a device, I can't see why it won't work.

The error#1 is:

out/soong/.bootstrap/bin/soong_build -t -l out/.module_paths/Android.bp.list -b out/soong
-n out -d out/soong/build.ninja.d -globFile out/soong/.bootstrap/build-globs.ninja -o
out/soong/build.ninja Android.bp

internal error: panic in WalkDeps(module "apex_test_build_features" variant "android_common_apex_test_build_features",

github.com/google/blueprint.(*baseModuleContext).WalkDeps.func1, ) for dependency module "prebuilt_libLLVM_android"
variant "android_arm_armv8-a_kryo_core_shared_apex_test_build_features" in GenerateBuildActions for module
"apex_test_build_features" variant "android_common_apex_test_build_features"

Requesting an invalid path

(full error log here)

Is it because I have to include libLLVM_android in some apex configuration so some checks for its inclusion to libart-compiler can now pass?

Or is it  that the prebuilt_libLLVM_android is missing?

On android6, I used to include libLLVM like this and it worked fine:

art/compiler/Android.mk:

...
    LOCAL_SHARED_LIBRARIES += libLLVM
    LOCAL_CFLAGS += -D__STDC_LIMIT_MACROS \
                     -D__STDC_CONSTANT_MACROS
    LOCAL_C_INCLUDES += $(LOCAL_LLVM_INCLUDES)


    ifeq ($$(art_target_or_host),target)
      # I think I am missing this part on Android.bp
      include $(LLVM_DEVICE_BUILD_MK)
    endif
    include $(LLVM_GEN_INTRINSICS_MK)
...

Thanks in advance,

Paschalis

[Paschalis Mpeis]

I've eventually made this work. My last issue was due to the target libLLVM_android being only available for host architectures (linux/darwin x86).

I made another target just for Android, and it did work, but since llvm version was too old (3.8 in particular), including those c++11 headers in art/compiler

have caused a lot of other issues. Making it another library and using that one from art/compiler would still wouldn't solve it.

Tinkering some art headers was messy, so I decided that the only solution was to use a more up-to-date llvm sources.

For this I've tried many many things, including:

1. cross compiling llvm, which worked but linking libLLVM.so against libart-compiler.so wouldn't, because Android NDK, has modified the std namespace. So I had unresolved symbols.

2. tried to compile NDK from sources to "work around" the namespace issue but I eventually gave up

3. tried to compile using the exact same flags that AOSP users for libart (except the c++11 one which LLVM requires), but stuck on issues as well.

I ended up doing what I've tried to avoid: integrated llvm9 sources into the soong build system as another external project, in a similar fashion with the outdated external/llvm.

To avoid collisions I named it external/llvmx (also for any blueprint targets as well), since the outdated llvm has dependencies with other parts of the AOSP.

I've created new blueprint targets (and referenced them in existing ones) for new code that were required for: libLLVM.so, llc, opt, and llvm-link.

The only difference from the external/llvm build process is that I didn't followed the same route for files generated by llvm's tblgen.

The tblgen dependencies have increased, and each of them required special handling (targets in .bp backed by some go lang scripts).

So I removed all the tblgen related targets from .bp files, and I'm including them as prebuilt headers from my cross-compiled version.

I can now link libLLVM.so against libart-compiler.so and actually use it. llvm9 headers do not cause any issues now.

Cheers,

Paschalis