BUILD SYSTEM INFORMATION
Code:
#### Build system information ####
exp@exp:~$
exp@exp:~$ sudo dmidecode -t system | grep "Manufacturer:\|Version:"
Manufacturer: LENOVO
Version: Lenovo Y50-70 Touch
exp@exp:~$
exp@exp:~$
exp@exp:~$
exp@exp:~$ sudo dmidecode -t processor | grep "Version\|Family:"
Family: Core i7
Version: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz
exp@exp:~$
exp@exp:~$
exp@exp:~$
exp@exp:~$ cat /proc/meminfo | grep MemTotal
MemTotal: 16332968 kB
exp@exp:~$
exp@exp:~$
exp@exp:~$
exp@exp:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.10
Release: 17.10
Codename: artful
exp@exp:~$
exp@exp:~$
BUILD SUMMARY
Code:
#### Total build time ####
38m6.816s
#### Build times ####
[GCC NDK r13b] : 4m15.596s
[GCC NDK r17] : 4m13.983s
[Android NDK r13b : LLVM/clang + binutils(ld and as)] : 4m3.665s
[Android NDK r17 : LLVM/clang + binutils(ld and as)] : 4m4.683s
[Main LLVM/clang + Android NDK r13b binutils(ld and as)] : 6m8.064s
[Main LLVM/clang + Android NDK r17 binutils(ld and as)] : 6m3.457s
[Qualcomm Snapdragon LLVM/clang + Android NDK r13b binutils(ld and as)] : 4m32.581s
[Qualcomm Snapdragon LLVM/clang + Android NDK r17 binutils(ld and as)] : 4m44.779s
LONGEST AND SHORTEST BUILD
Code:
##### Longest build ####
Name : Main LLVM/clang + Android NDK r13b binutils(ld and as)
Time : 6m8.064s
boot.img : boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img
zImage-dtb : zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as
##### Shortest build ####
Name : Android NDK r13b : LLVM/clang + binutils(ld and as)
Time : 4m3.665s
boot.img : boot-ndk-r13b-clang-llvm-binutils-ld-as.img
zImage-dtb : zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as
LARGEST AND SMALLEST IMAGES
Code:
◈ Largest boot img : boot-ndk-r17-gcc.img ❏ Size : 13M (12984320 bytes)
◈ Smallest boot img 1 : boot-main-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size : 11M (11272192 bytes)
◈ Smallest boot img 2 : boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size : 11M (11272192 bytes)
◈ Largest zImage-dtb : zImage-dtb-ndk-r17-gcc ❏ Size : 12M (11844904 bytes)
◈ Smallest zImage-dtb : zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size : 9.7M (10132600 bytes)
BOOT IMAGES SUMMARY
Code:
◈ boot-ndk-r17-gcc.img ❏ Size ~ 13M (12984320 bytes)
◈ boot-ndk-r13b-gcc.img ❏ Size ~ 13M (12978176 bytes)
◈ boot-ndk-r13b-clang-llvm-binutils-ld-as.img ❏ Size ~ 13M (12625920 bytes)
◈ boot-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size ~ 12M (11610112 bytes)
◈ boot-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size ~ 12M (11610112 bytes)
◈ boot-ndk-r17-clang-llvm-binutils-ld-as.img ❏ Size ~ 11M (11476992 bytes)
◈ boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size ~ 11M (11272192 bytes)
◈ boot-main-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size ~ 11M (11272192 bytes)
ZIMAGE-DTB SUMMARY
Code:
◈ zImage-dtb-ndk-r17-gcc ❏ Size ~ 12M (11844904 bytes)
◈ zImage-dtb-ndk-r13b-gcc ❏ Size ~ 12M (11837640 bytes)
◈ zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as ❏ Size ~ 11M (11487176 bytes)
◈ zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size ~ 10M (10469728 bytes)
◈ zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as ❏ Size ~ 10M (10469680 bytes)
◈ zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as ❏ Size ~ 10M (10336624 bytes)
◈ zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as ❏ Size ~ 10M (10132608 bytes)
◈ zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size ~ 10M (10132600 bytes)
RAMDISK INFORMATION
Code:
Ramdisk(pre-built - from RR) :
◈ boot.img-ramdisk.gz ❏ Size ~ 1M (1136400 bytes)
Clang-KERNEL INFORMATION(from each of the zImage-dtb images)Code:
exp@exp:~$
exp@exp:~$ ../show_kernel_compiler_all.sh
#### Kernel compiler information ####
NOTE : Analyzing the images by decompressing them based on lz4 magic (\x02\x21\x4c\x18) . . .
Image metadata(zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as) :
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5381
++ dd if=zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as bs=5381 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:33:21 PDT 2018
++ set +x
Image metadata(zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as) :
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as
+++ cut -d: -f 1
+++ tail -1
++ pos1=5373
++ dd if=zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as bs=5373 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:39:25 PDT 2018
++ set +x
Image metadata(zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as) :
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5509
++ dd if=zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as bs=5509 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ grep 'clang version'
+++ strings -a
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android clang version 3.8.256229 (based on LLVM 3.8.256229) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:23:08 PDT 2018
++ set +x
Image metadata(zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as) :
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5449
++ dd if=zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as bs=5449 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android (4691093 based on r316199) clang version 6.0.2 (https://android.googlesource.com/toolchain/clang 183abd29fc496f55536e7d904e0abae47888fc7f) (https://android.googlesource.com/toolchain/llvm 34361f192e41ed6e4e8f9aca80a4ea7e9856f327) (based on LLVM 6.0.2svn) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:27:14 PDT 2018
++ set +x
Image metadata(zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as) :
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5417
++ dd if=zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as bs=5417 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:43:58 PDT 2018
++ set +x
Image metadata(zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as) :
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5409
++ dd if=zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as bs=5409 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:48:42 PDT 2018
++ set +x
exp@exp:~$
exp@exp:~$
Clang-KERNEL INFORMATION(from dmesg extracted from each of the boot instances)
Code:
exp@exp:~$
exp@exp:~$ cat android1/android1_dmesg.txt | grep "clang\|Machine"
[ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android clang version 3.8.256229 (based on LLVM 3.8.256229) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:23:08 PDT 2018
[ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$
exp@exp:~$
exp@exp:~$ cat android2/android2_dmesg.txt | grep "clang\|Machine"
[ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android (4691093 based on r316199) clang version 6.0.2 (https://android.googlesource.com/toolchain/clang 183abd29fc496f55536e7d904e0abae47888fc7f) (https://android.googlesource.com/toolchain/llvm 34361f192e41ed6e4e8f9aca80a4ea7e9856f327) (based on LLVM 6.0.2svn) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:27:14 PDT 2018
[ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$
exp@exp:~$
exp@exp:~$ cat main1/main1_dmesg.txt | grep "clang\|Machine"
[ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:33:21 PDT 2018
[ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$
exp@exp:~$
exp@exp:~$ cat main2/main2_dmesg.txt | grep "clang\|Machine"
[ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:39:25 PDT 2018
[ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$
exp@exp:~$
exp@exp:~$
exp@exp:~$ cat qualcomm1/qualcomm1_dmesg.txt | grep "clang\|Machine"
[ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:43:58 PDT 2018
[ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$
exp@exp:~$
exp@exp:~$ cat qualcomm2/qualcomm2_dmesg.txt | grep "clang\|Machine"
[ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:48:42 PDT 2018
[ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$
Finding the right kernel source and kernel config that works with the Android version I had on my Nexus 5.
Android(version) on my Nexus 5 is Resurrection Remix(RR) oreo with 3.4.13 hammerhead kernel(unicornblood config)
Experimenting with different kernel source and hammerhead kernel config including that of AOSP
Finding the most compact and quickest way to just build the kernel out of tree but using the tree's build tools
Syncing the Resurrection Remix's source from its repo onto my machine
Finding the make targets for building just the ramdisk instead of building the whole RR ROM which was not the goal
Finding the tool and the arguments for that to build the boot.img that can be fastboot-ed
Building a working kernel from source for my RR on Nexus 5, first with Android NDK r13b gcc
Finding the DirtyUnicorn kernel source, building it with gcc in the first place, generating the boot.img
Finding the right kernel config for the hammerhead kernel - unicornblood that's used by DirtyUnicorn repo
Using the pre-built ramdisk from RR zip instead of self-built one
Disabling SELinux for allowing the kernel to be fastboot-ed
Debugging the ADB-over-USB not working with the DirtyUnicorn kernel image built
Discussing on the Resurrection Remix forum and with the kernel developer(uname:voidz) to know the actual kernel source used in RR.
Successfully booting the gcc-built kernel for RR on my Nexus 5
Setting up the clang-kernel build
Finding the right LLVM/clang toolchain to begin with - Android NDK r13b LLVM/clang
Using the binutils - assembler, linker, etc that's in Android NDK r13b
Launching the clang-kernel build
Disabling the options that clang doesn't recognize
Examining the initial compilation erros - invalid instruction error(thrown actually by x86_64 GNU as)
Fixing the assembler path to the ARM assembler
Making sure it uses the right external assembler(Android EABI GNU assembler)
Fixing the subsequent build errors
Updating VLAIS to non-VLAIS in various kernel components
Fixing linker errors for duplicate exception sections that clang generates
Fixing more linker errors
Booting the clang-built kernel image for the first time
Booting gets stuck at "Google" logo
Checking whether there was any kernel panic by booting to recovery mode and checking /proc/last_kmsg
Not finding anything relevant to clang-kernel boot in /proc/last_kmsg
Running over a plethora of possibilities for the stuck-at-google-logo case
Kernel might have not been loaded at all by bootloader for some reason
Generated boot.img might have incorrect offsets for kernel, ramdisk, etc
Generated kernel image might be corrupt
Bootloader might be expecting gcc-specific compiler metadata instead of LLVM/clang, in kernel image header
Fastboot might have some way of logging the overall boot sequence which might have give some hint
Attempting to boot the clang-kernel and ramdisk with QEMU/ARM to debug like I did for x86_64 clang-kernel
Booting on QEMU/ARM with both kernel and ramdisk, next with just the kernel, with GUI and without GUI
Booting the kernel and ramdisk built with Android emulator, specifically QEMU/ARMel that's part of the SDK
Cross-compiling Android LittleKernel bootloader for ARM and using that to boot the clang-kernel on QEMU/ARM
Not seeing anything happening at all in any of the above scenarios
Carrying out more debugging
Turning off the Nexus 5 and doing a fastboot to get fresh /proc/last_kmsg if possible
Adding various debug parameters on kernel command-line
Trying out different options that fastboot has for specifying offsets, etc
Researching on external hardware-based debugging like adding UART interface to get early boot logs
Researching on the very first code that runs when the kernel is loaded - kernel entry point
Finding start_kernel() inside init/main.c
Adding some debug statements over there and not seeing them for obvious console-not-yet-initialized reason
Then finding the actual kernel entry point in head.S assembly source
Researching on ways to print anything at all in the ARM assembly code within head.S
Finding printascii that's used for the above case but realizing it's for a serial console(UART)
Understanding the ARM assembly code within head.S and its siblings and the inline documentation therein
Exploring the methods to confirm whether the control is indeed reaching head.S or not
Checking if the LED on the bottom of Nexus 5 can be turned on/off with different colors as an indication
Researching on doing something with ARM CPU like raising an exception, or a reset event as an indication
Researching on different ways of restarting Nexus 5 - checking how a "reboot" command works at kernel level
Looking into machine restart logic - translating that to a reset logic to be used within head.S
Using reset logic for more fine-grained debugging instruction by instrucion within head.S and its siblings
Noticing PC write was problematic
Finding ways to branch off to destination instead of modifying PC which is not recommended as per the docs
Understanding the end-to-end control flow since the kernel entry point till start_kernel() of init/main.c
Following the inter-working of head.S and processor-specific assembly code during the setup
Locating some control register access being problematic like that with PC
Researching on other available clang/LLVM toolchains to see if they work
Using Android NDK r17's LLVM/clang - not helpful - same outcome - stuck-at-google-logo case
Finding main LLVM/clang source and building it to use with kernel source - not helpful - same outcome
Finding Snapdragon Qualcomm LLVM/clang toolchain and using it - not helpful - same outcome
Using different diff-tools to compare two binaries : clang-built kernel and gcc-built kernel
Using different binary analysis tools to examine the differences between the gcc-built and clang-built kernels => Android EABI readelf - saw some ELF header information differences
Adding mechanism to use the same assembly code settings as that of gcc
Using same assembler options that gcc uses while invoking the assembler for intermediate assembly code
Using same assembly code setup as that of gcc for data, target architecture, etc for intermediates
Automating the above so that it works for every intermediate assembly file that clang generates
Reducing the optimization level of kernel build
Keeping the oversmart optimization aside - O1 and Os - didn't change the stuck-at-google-logo case
Disabling optimization completely - O0 - didn't work - kernel doesn't support based on what I read online
Disabling the caches in kernel config as needed by head.S
Trying out different assembler options
Trying out different clang options
Correcting the SP access
Researching more on the lines on what does clang do that's not gcc doesn't
Disabling all the optimizations and clang-only features if any
Disabling all the intrinsic features that clang uses internally - device rebooted after a failed boot!
No more stuck-at-google-logo case with the above change
Witnessing the device auto-reboot with the above change - must be a kernel panic!
Checking /proc/last_kmsg within recovery mode - yes, it was the clang-kernel that panic'ed - good sign!
Finally, the clang-kernel has started executing after an exhaustive set of attempts - breakthrough!
Locating the source of kernel panic
Looking at the stacktrace revealed one of the audio codec had a buffer overrun - fixed it
Rebuilding the kernel with the fix and retrying
Seeing some more kernel panics - another audio codec source which had similar issue - fixed it
Booting to the Android GUI with clang-kernel for the first time
Fixing the kernel panics mentioned above allowed boot to move on
Seeing Android animation(RR logo in my case) for the first time!
Getting to the Android home screen after few seconds of wait - mission accomplished!
Verifying all the system information
Checking kernel version to be showing LLVM/clang toolchain version, etc
Examining kernel dmesg for clang specific information
Checking /proc/version for the same
Checking Settings/About for the same
Verifying all the features work
Checking Camera, Bluetooth, ADB over USB, etc
Cheking WiFi - didn't work - "connected, no internet"
Noticing WiFi symbol had a cross(x) symbol on it
Noticing mobile network also had a cross(x) symbol on it after disabling WiFi as above
Browsing with mobile network as well failed as expected due to no internet availability
Verifying phone calls work - yes, they worked!
Checking logcat, dmesg for any network error
Noticing SELinux denials for some of the network related actions
Locating the error stating bandwidth module not loaded
Narrowing down to the kernel code where the possible issue is present
Fixing the netfilter code for the above issue
Realizing all the features are now working with a clang-built ARM kernel for Android!
Using Android NDK r13b's LLVM/clang in place of main LLVM/clang used so far for building the kernel
Fixing the Camera MSM driver code in terms of device id specification
Comparing with other Camera MSM driver source code and finding the difference if any
Completing the device/driver id specification with the missing item - fixed the panic
Booting to Android home screen this time with even the Android NDK r13's LLVM/clang-built kernel!
Picking the next remaining LLVM/clang toolchains
Performing round up of all the toolchains and combinations of NDK binutils
Automating all of the above builds and the testing of the images
Facilitating automation of all the 8 combination builds
Collecting the statistics - build time, image sizes, etc
Summarizing the longest/shortest builds, largest/smallest zImage-dtb, largest/smallest boot.img
Testing all of the images one by one for the final time
Consolidating the data from the above automation
Collecting the complete kernel boot(dmesg) log from each build
Taking snapshots of the kernel version, Android version, build number from each build - About/System info
Wrapping my Android clang-kernel research project!