No response after “lunch" command

[Alessandro Brucato]

Hello, I'm trying to build the Android source code for my Hikey 960 board.
I'm following this video tutorial https://www.youtube.com/watch?v=RD_NZRKiZBU in parallel to the official Android guide https://source.android.com/setup/devices#hikey-boards.

Everything went fine until I reached the command "lunch hikey960-userdebug". It blocks, without returning any output. I left it "working" for 1 hour and then I aborted it (even if I know that this command should take few seconds). If I execute only "lunch" it shows the list of available build targets, and when I select "hikey960-userdebug" it blocks. When I abort it with CTRL+C, it says "Still alive, killing subprocesses" and then it shows a lot of messages referring to threads. Here is the screenshot https://ibb.co/k32KQR
I noticed that also the "make" command (inside the AOSP directory) has the same behaviour.

This is the list of commands I've used during the process:

sudo apt-get install openjdk-7-jdk

sudo apt-get install git-core gnupg flex bison gperf build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache libgl1-mesa-dev libxml2-utils xsltproc unzip

mkdir output

export OUT_DIR_COMMON_BASE=/home/brucedh/output/

mkdir ~/bin

PATH=~/bin:$PATH

curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo

chmod a+x ~/bin/repo

mkdir dev

cd dev

mkdir hikey960

cd hikey960

git config --global user.name "<REDACTED>"

git config --global user.email "<REDACTED>"

repo init -u https://android.googlesource.com/platform/manifest -b master

repo sync –j4

git clone https://android.googlesource.com/kernel/hikey-linaro

cd hikey-linaro

git checkout -b android-hikey-linaro-4.9 origin/android-hikey-linaro-4.9

make ARCH=arm64 menuconfig

make ARCH=arm64 hikey960_defconfig

PATH=/home/brucedh/dev/hikey960/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/:$PATH

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- -j4

cp arch/arm64/boot/dts/hisilicon/hi3660-hikey960.dtb ../device/linaro/hikey-kernel/hi3660-hikey960.dtb-4.9

cp arch/arm64/boot/Image.gz ../device/linaro/hikey-kernel/Image.gz-hikey960-4.9

cd ~/dev/hikey960

. ./build/envsetup.sh

lunch hikey960-userdebug

I'm using Ubuntu 16.04 LTS as guest on Virtualbox. I tried to do all the process once again from the beginning with a new virtualbox image, but I found the same problem.

Thanks in advance! 

[Colin Cross]

Can you capture the full output of all the stack traces?

It's getting stuck while trying to walk all the checked-out code looking for various build files.  If it was completely deadlocked then it would have died on its own, so it's probably stuck in a loop somehow.

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-building@googlegroups.com
To unsubscribe from this group, send email to
android-building+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Warrington]

Hello, I've had this same problem building for a different target too. When I press ctrl+c to exit I got the following output repeated:

goroutine 6171 [chan send]:
android/soong/finder.(*semaphore).Lock(...)
    build/soong/finder/finder.go:589
android/soong/finder.(*threadPool).Run.func1(0xc4200f0500, 0xc420f5e720)
    build/soong/finder/finder.go:626 +0x41
created by android/soong/finder.(*threadPool).Run
    build/soong/finder/finder.go:625 +0x62

This seems to suggest that it is having trouble finding the files on my external HDD. A workaround of this in theory would be to disable "Soong" which replaces makefiles (See https://android.googlesource.com/platform/build/soong/) but I do not know if this is possible or how to do it.

To post to this group, send email to android-...@googlegroups.com

To unsubscribe from this group, send email to

android-buildi...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-buildi...@googlegroups.com.

[Alessandro Brucato]

This is the output of strace attached to the process in which I execute "lunch".

When "lunch" is stuck, the output of strace ends with "wait4(-1,". All the remaining part appears when I abort "lunch" with CTRL+C.

Maybe this issue is due to the fact that I'm using a guest in Virtualbox?

To post to this group, send email to android-...@googlegroups.com

To unsubscribe from this group, send email to

android-buildi...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-buildi...@googlegroups.com.

[Colin Cross]

There will be at least one goroutine with a different stack.  Can you capture the full log and put it somewhere I can see?

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.

[Colin Cross]

From the stack trace after ctrl-c, lunch will be waiting for the child soong_ui process to complete.  An strace of that process would be interesting.

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.

[Peter Warrington]

Attached is a partial strace.

[Colin Cross]

Nothing looks wrong in that strace, it is making forward progress.  It needs to read every directory and stat every file in the tree, is your filesystem very slow for some reason?

One difference I see between the old implementation using the find tool and the new implementation is that find seems to be able to avoiding calling stat on every directory entry, because the relevant information (whether the entry is a subdirectory or a file) is contained in the return from the getdents64 syscall.  The Go implementation is ignoring that information and has to call lstat to get it.

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.

[Peter Warrington]

Thank you! That is very interesting. In case it helps, I benchmarked my external HDD and it has an average read rate of 92.0 MB/s and an average access time of 16.37 msec. However, I didn't have any trouble with this previously.

[Colin Cross]

Can you attach a longer strace with -T ?

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.

[Peter Warrington]

Hello, the command has now started to work like normal after leaving it for a while, so my problem might not be the same as Alessandro's. Unfortunately, I cannot upload a bigger stack trace until tomorrow because of the time in my timezone. Sorry!

[Colin Cross]

The results of the finder are cached in out/.module_paths/files.db, so once the command has completed successfully once the next runs will be much faster.  You could try wiping that file to reproduce it again, or run with a different value for OUT_DIR (OUT_DIR=out2 lunch ...).

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.

[Peter Warrington]

Attached is a much longer strace with the -T option used.

[Colin Cross]

I don't see the data I'd expect with -T, it should have a time in angle brackets after each finished syscall:

[pid 219359] <... lstat resumed> {st_mode=S_IFREG|0640, st_size=323161, ...}) = 0 <0.000046>

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.

[Alessandro Brucato]

First of all, thanks for all the answers.

Here is my strace with -T option:

strace: Process 3045 attached

wait4(-1, 0x7ffc9f0c66c0, 0, NULL)      = ? ERESTARTSYS (To be restarted if SA_RESTART is set) <1.358891>

--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---

rt_sigreturn({mask=[CHLD]})             = -1 EINTR (Interrupted system call) <0.000019>

wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 2}], 0, NULL) = 3046 <5.282143>

rt_sigaction(SIGINT, {0x45f790, [], SA_RESTORER, 0x7f3bb53244b0}, {0x4449b0, [], SA_RESTORER, 0x7f3bb53244b0}, 8) = 0 <0.000004>

rt_sigprocmask(SIG_BLOCK, [CHLD TSTP TTIN TTOU], [CHLD], 8) = 0 <0.000003>

ioctl(255, TIOCSPGRP, [2426])           = 0 <0.000005>

rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0 <0.000003>

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000003>

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3046, si_uid=1000, si_status=2, si_utime=72, si_stime=255} ---

wait4(-1, 0x7ffc9f0c6110, WNOHANG, NULL) = -1 ECHILD (No child processes) <0.000004>

rt_sigreturn({mask=[]})                 = 0 <0.000004>

exit_group(2)                           = ?

+++ exited with 2 +++

[Peter Warrington]

Sorry about that. Is this attached strace any better?

[Colin Cross]

Yeah, this is showing very slow lstat calls:

[pid  4007] <... lstat resumed> {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0 <0.020417>

vs. mine:

[pid 219359] <... lstat resumed> {st_mode=S_IFREG|0640, st_size=323161, ...}) = 0 <0.000046>

I don't know why yours would be so much slower, maybe just round trip latency to the external drive and not enough memory to keep them all in the inode cache?

We can improve finder to drastically reduce the number of lstat calls, but ninja is going to have to stat most of the files to determine what needs building so I don't know how much improvement you will see.

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.

[Peter Warrington]

Thank you for all your help and good luck with improving finder! With regards to memory, I have a measly 4GB of ram so that is probally slowing finder down and I do expect it to be slow (I do plan on upgrading in case you are wondering). However, I do not believe what I am experiencing is the same as what Alessandro is experiencing. Again, Thanks for all your help!

[Glenn Kasten]

Sorry to hear about that.  4G is not nearly enough RAM to be able to build successfully.  You need much much more, as much as you can possibly get.

[Colin Cross]

The patch stack at https://android-review.googlesource.com/c/platform/build/soong/+/575773 removes >700k lstat calls from finder, I'm curious if it makes any difference for you.

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.

[Peter Warrington]

Thank you! It cuts about a quater off the time finder normally takes.

[Elizabeth Camarena]

To post to this group, send email to android-building@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscribe@googlegroups.com.