Segmentation Faults During Builds

[Tom Deering]

Hello Android Wizards,

I am trying to build Android and have followed the instructions on the 'Downloading and Building' page. No matter which branch or configuration I build, or whether I used Ubuntu 12.04 or 14.04, I encounter the same problem. Deep into the build, gcc always has an internal segmentation fault, stopping all progress before the build can complete.

Does everyone else encounter gcc segfaults when building Android? If so, how do you deal with them? Thanks in advance.

[allicoder redocilla]

I've never seen this happening before. Does it always happen at the same file? What is your gcc version (output of $ gcc --version)? Is this with completely clean checkouts? Do you have parallel builds enabled?

alli

[Tom Deering]

I've got gcc 4.8.2:

> gcc --version

gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2

I think I've seen it happen on multiple files, though something in the Chromium project seems particularly good at triggering it. I'll double-check. Yes, the checkouts of Android are completely clean (I've made no changes). Yes, I'm using parallel builds.

Tom Deering | PhD Student | ISU ECpE Software Systems | (563) 249-9277

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to
android-buildi...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-buildi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Deering]

Ok, I just ran a build and produced an example of what I am seeing: 

>make -j16

============================================

PLATFORM_VERSION_CODENAME=REL

PLATFORM_VERSION=4.4.3

TARGET_PRODUCT=full

TARGET_BUILD_VARIANT=eng

TARGET_BUILD_TYPE=release

TARGET_BUILD_APPS=

TARGET_ARCH=arm

TARGET_ARCH_VARIANT=armv7-a

TARGET_CPU_VARIANT=generic

HOST_ARCH=x86

HOST_OS=linux

HOST_OS_EXTRA=Linux-3.13.0-32-generic-x86_64-with-Ubuntu-14.04-trusty

HOST_BUILD_TYPE=release

BUILD_ID=KTU84L

OUT_DIR=out

============================================

...

external/sqlite/dist/sqlite3.c: In function 'sqlite3VXPrintf':

external/sqlite/dist/sqlite3.c:19527:21: internal compiler error: Segmentation fault

Please submit a full bug report,

with preprocessed source if appropriate.

See <http://gcc.gnu.org/bugs.html> for instructions.

target thumb C++: libandroidfw <= frameworks/base/libs/androidfw/StreamingZipInflater.cpp

make: *** [out/host/linux-x86/obj/SHARED_LIBRARIES/libsqlite_intermediates/sqlite3.o] Error 1

Tom Deering | PhD Student | ISU ECpE Software Systems | (563) 249-9277

[Al Sutton]

Information about your hardware would help. If you’re doing -j16 I’d expect you to have 8+ cores and probably 20+GB of RAM to be safe.

Al.

-- 
Al Sutton - Funky Android Ltd.
(funkyandroid.com)

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

[Jean-Baptiste Queru]

Those typically indicate a RAM problem, i.e. a faulty chip, poor cooling, or overclocked hardware, especially when using non-ECC RAM. gcc pushes RAM especially hard and it is known to trigger RAM errors on hardware that otherwise seems rock solid.

JBQ

 

--

Jean-Baptiste M. "JBQ" Quéru
Architect, Mobile, Yahoo

[Tom Deering]

My laptop has a 4-core, 8-thread Intel Core i7-3610QM, and 16GB of DDR3. I ran with -j16 to keep all cores busy even when some jobs were blocking for disk I/O. I've put the ram through memtest86+ several times and never observed any errors. Is that sufficient to eliminate that hypothesis? I should mention... if I run the build repeatedly, it does eventually succeed. That indicates to me that somehow the failure is non-deterministic or intermittent. This non-determinism occurs even with -j1.

Ideas for what to investigate next?

Tom Deering | PhD Student | ISU ECpE Software Systems | (563) 249-9277

[Jean-Baptiste Queru]

I've seen at least one machine that passed memtest but still crashed in gcc. It turned out to be a bad motherboard, where one of the RAM slots didn't work well enough.

JBQ

--

Jean-Baptiste M. "JBQ" Quéru
Architect, Mobile, Yahoo

[Al Sutton]

Does the laptop get warm when building? 

I don’t think memtest stresses the CPU and the RAM at the same time, so it could be CPU+RAM heat which is going beyond what your machine can take, and if you have a HDD rather than an SSD that will add to the problem.

Al.

-- 
Al Sutton - Funky Android Ltd.
(funkyandroid.com)

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

[Tom Deering]

My laptop's got an SSD rather than HDD, but you two could be right about the CPU+RAM or ram slot combination. I'll see if Prime95 or something can show problems.

Would it make any sense for the SSD to be responsible? I recently got a new SSD. Perhaps intermediate binaries and libraries were corrupted as they were written to disk, resulting in later errors as they were used?

The laptop gets warm but not hot when building. The fan heads up to about its 80% speed/loudness.

Tom Deering | PhD Student | ISU ECpE Software Systems | (563) 249-9277

[Jean-Baptiste Queru]

If the issue was related to the SSD holding bad data, the crashes would be more consistent since they'd be related to a persistent state (and you'd probably see fewer crashes as gcc does a good job at detecting bad input).

If the issue was related to the SSD holding correct data but bad data being read in, you'd probably see crashes or issues in javac as well, since that also reads a whole lot of data (more than gcc in fact).

JBQ

 

--

Jean-Baptiste M. "JBQ" Quéru
Architect, Mobile, Yahoo

[Tom Deering]

Good to know. The SSD passed an extensive SMART test anyway. Prime95 didn't turn up any errors in any of its three stress test modes either. Not sure what to say.

This certainly reeks of a hardware problem, even though I can't demonstrate it. I figured there was no way it could be a widespread problem affecting many devs, or else there'd be hoards of unhappy people. Thank you for your time and responses. For now I'm inclined to shrug and use a different machine, unless you have other suggestions.

Tom Deering | PhD Student | ISU ECpE Software Systems | (563) 249-9277

[Jean-Baptiste Queru]

On a laptop, you might have opportunities to force the system to lower power modes, especially reduced clock speeds, though there's a risk that those would imply lower voltages which in turn could make things worse.

It's hard to imagine that you'll find options that make the build significantly more reliable without also deeply slowing it down.

JBQ

 

--

Jean-Baptiste M. "JBQ" Quéru
Architect, Mobile, Yahoo

[Tom Deering]

I tried to give my laptop some tender loving care by cleaning out dust, upgrading thermal paste between the heat sink and cpu/gpu, and re-seating the ram. No luck on a subsequent build. This time it was g++ which died, not gcc. :(

> make -j10

============================================

PLATFORM_VERSION_CODENAME=REL

PLATFORM_VERSION=4.4.3

TARGET_PRODUCT=full

TARGET_BUILD_VARIANT=eng

TARGET_BUILD_TYPE=release

TARGET_BUILD_APPS=

TARGET_ARCH=arm

TARGET_ARCH_VARIANT=armv7-a

TARGET_CPU_VARIANT=generic

HOST_ARCH=x86

HOST_OS=linux

HOST_OS_EXTRA=Linux-3.13.0-32-generic-x86_64-with-Ubuntu-14.04-trusty

HOST_BUILD_TYPE=release

BUILD_ID=KTU84L

OUT_DIR=out

============================================

...

target thumb C++: sandbox_sandbox_services_gyp <= external/chromium_org/sandbox/linux/services/init_process_reaper.cc

arm-linux-androideabi-g++: internal compiler error: Segmentation fault (program as)

Please submit a full bug report,

with preprocessed source if appropriate.

See <http://source.android.com/source/report-bugs.html> for instructions.

make: *** [out/target/product/generic/obj/STATIC_LIBRARIES/base_base_prefs_gyp_intermediates/base/prefs/pref_service_factory.o] Error 4

make: *** Deleting file `out/target/product/generic/obj/STATIC_LIBRARIES/base_base_prefs_gyp_intermediates/base/prefs/pref_service_factory.o'

make: *** Waiting for unfinished jobs....

It seems very clear now that it *must* be a hardware problem. In fact, some other software on my system has been a bit unstable as well. I definitely want to diagnose and fix this, but that's an irritating task when I can't pin down the problem. I know it's not the purpose of this group, but do you have suggestions for how to proceed in debugging flaky hardware? Given that memtest and prime95 were unhelpful, I was thinking of trying another set of ram and seeing if the problem goes away. If it does, the ram must be to blame. If it doesn't... the mobo or cpu are probably bad, suggesting I should simply replace the laptop.

Any other thoughts? If not, I'll stop asking :)

[Tom Deering]

Does "Bus error" mean anything to anyone? To me, it sounds like it implicates the memory bus on the motherboard.

In file included from external/chromium_org/v8/src/v8.h:52:0,

                 from external/chromium_org/v8/src/assembler.h:38,

                 from external/chromium_org/v8/src/assembler.cc:35:

external/chromium_org/v8/src/../include/v8.h:3233:8: internal compiler error: Bus error

Please submit a full bug report,

with preprocessed source if appropriate.

See <http://source.android.com/source/report-bugs.html> for instructions.

make: *** [out/target/product/generic/obj/STATIC_LIBRARIES/v8_tools_gyp_v8_base_arm_gyp_intermediates/v8/src/assembler.o] Error 1

make: *** Waiting for unfinished jobs....

Tom Deering | PhD Student | ISU ECpE Software Systems | (563) 249-9277

You received this message because you are subscribed to a topic in the Google Groups "Android Building" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-building/p7_JAKSpO1M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to android-buildi...@googlegroups.com.

[Tom Deering]

Disregard my last. Looks like a bus error is another kind of hardware fault which is a lesser-known cousin of the seg fault.

http://en.wikipedia.org/wiki/Bus_error

Tom Deering | PhD Student | ISU ECpE Software Systems | (563) 249-9277

[Tom Deering]

I think I'm all set. I tested my ram sticks individually in each slot using the build.

        | Slot 1 | Slot 2 |

Stick 1 |  Fail  |  Fail  |

---------------------------

Stick 2 |  Pass  |  Pass  |

It would seem that stick 1 is bad. Hopefully this helps someone in the future, and is evidence that builds may fail even though memtest passes!

Tom Deering | PhD Student | ISU ECpE Software Systems | (563) 249-9277

[mahe...@samsung.com]

I'm recently facing the same Bus error and Seg faults frequently on my linux machine during chromium builds.
My machine is a Dell desktop with core i7 vPro and having 4 sticks of RAM (4+4+8+8).
Can you please help me- how to find out which RAM is faulty while running the build?