Re: Inquiry on running KASAN enabled 3.18 kernel on MSM devices (e.g. Pixel)

[Dmitry Vyukov]

On Sat, Dec 31, 2016 at 5:23 AM, edward yuki <edward...@gmail.com> wrote:
> Hi Dmitry:
>
> Thanks for the great work on KASAN! I tried to enable KASAN on MSM kernels
> and ran it on Pixel devices, but it seems the kernel won't boot, and
> unfortunately even no console-ramoops is produced to help debugging this
> issue. (Normally if there's a kernel panic during boot, MSM devices will
> produce console-ramoops and I can retrieve this log to aid debugging).
>
> I wonder if there's some advice on running KASAN kernel builds on MSM
> devices? Is there someone I can contact for help also?
>
> Thanks very much and wish you a happy new year!


HI Edward,

I did not build Pixel/KASAN kernels.
+More people who should know how to do it.

Thanks

[Billy Lau]

I am unsure of the difference between the kernel config that was applied that made KASAN work internally vs the one we have published in AOSP though. But by the sound of the symptoms that was described, it seemed that you might have run into the case where KASAN kernel has a size that is too large to be loaded by the bootloader - therefore the kernel doesn't even boot and therefore no console-ramoops is produced. We overcome this internally by changing certain values in the device's boardconfig.

Billy Lau |

Android Attacker |

bill...@google.com |

A Galaxy Far, Far Away

[zhangbo...@gmail.com]

Hi Billy,

I'm sorry to bother you. I encounter the same problem: i enabled KASAN on android kernel 3.18 and ran it on pixel, but the phone boot failed. the symptoms is very similar to what edward described.

We overcome this internally by changing certain values in the device's boardconfig.

Could you share your method to make kasan working? I will be very grateful. thank you.

here is my probelm:  https://groups.google.com/forum/#!topic/syzkaller/gIj8UrgPxAY

I'm working on android kernel 3.18, when i enable KASAN, the phone boot failed and  went into bootloops; if disabling KASAN, it works fine.

I have only a few logs via a unstable serial cable, it provides little infomation, It seems that the kernel failed in a very early stage, maybe in the bootloader.

I have no idea about how to make it work, does anyone enabling kasan on android kernel successfully? 

if any suggestion, i will be very grateful.

here is some logs:

[840] pmi8994_is_charger_usbin_suspend: 0x807=0xC0, 0x1242=0x0, 0x1309=0x2, 0x1310=0x4, 0x1340=0x1, 0x1610=0x5

[850] pm8x41_get_is_cold_boot: cold boot

[860] check_reboot_mode: restart_reason: 0x00000000

[860] target_pause_for_battery_charge : pon_reason is 49 cold_boot:1

[870] target_pause_for_battery_charge : GCC_RESET_STATUS(0x66BF028): 0x2 

[870] LED off !!

[880] [multislot] BOOT_LUN = 2 

[880] [multislot] attribute:boot_a=0000009b

[880] [multislot] attribute:boot_b=00000087

[890] pm8x41_get_is_cold_boot: cold boot

[890] [DISP] Trigger updating for MIPI_CMD_PANEL.

[910] Backlight enable = 1 

[920] [DISP] Trigger updating for MIPI_CMD_PANEL.

R*

  UH.KWW/$TSŁL[5200] htc_mmc_read_alignment: 'ramdumHKWW/$R/&LLM&&LW_mmc_read_alignment: 'ramdump', transform (0x20a800, 458752) -> (0x20a000, 4628486250] Add DTB entry 420/00000003/0x000 Android Bootloader - UART_DM Initialized!!!

// here the phone restart

  [0] welcome to lk

在 2017年1月3日星期二 UTC+8下午9:12:46，Billy Lau写道：

[Billy Lau]

Hello there,

I am currently out of office. But prior to this, I have prepared a document that is currently in the pipeline to be published as a blog post outlining exactly this issue, on how to get a KASAN build for Pixel devices, which should hopefully be out in a few weeks time. In the meantime, I appreciate your patience on this.

- billy

[zhangbo...@gmail.com]

Billy, Thank you for your reply. I'm looking forwad to your blog post.

I think it's a common issue. I compiled the latest android oreo kernel(3.18) with kcov, run it on Pixel, the phone also boot failed. 

Bootloader prints log: decompress the lz4 kernel, then restarts the phone. The kernel with kcov become a little larger than orignal, i guess it's the reason.

Thanks.

在 2017年8月24日星期四 UTC+8上午1:46:28，Billy Lau写道：

[Billy Lau]

On Mon, Aug 28, 2017 at 12:12 AM, <zhangbo...@gmail.com> wrote:

Billy, Thank you for your reply. I'm looking forwad to your blog post.

I think it's a common issue. I compiled the latest android oreo kernel(3.18) with kcov, run it on Pixel, the phone also boot failed. 

Bootloader prints log: decompress the lz4 kernel, then restarts the phone. The kernel with kcov become a little larger than orignal, i guess it's the reason.

To overcome this, you can deploy these 2 strategies:

1. Not compress the kernel with lz4, but with gzip, by removing the LZ4 related option during kernel compilation.

2. Make sure that the kernel boots from a different offset - please refer to the parameters set in device/google/marlin/marlin/BoardConfig.mk defined specifically for marlin_kasan conditions.

Hope that helps.

Lastly, the document I mentioned earlier is in its final stage of editing/approval. Thanks for your patience.

- billy

[dar...@gmail.com]

Hi Billy, I am having the same exact issue. However I am compressing with GZIP and not LZ4. Any suggestions? It is straight up the marlin_defconfig with KCOV, KASAN, SLUB_DEBUG, and DEBUG_INFO enabled.

[dar...@gmail.com]

An update, I have finally ripped out everything imaginable that will still allow the phone to boot (even removed some stuff from the ramdisk). Now when I build the kernel with KCOV, KASAN, SLUB_DEBUG, and DEBUG_INFO it does not boot. I have also tried changing the offsets in the BoardConfig.mk file

[dar...@gmail.com]

Sorry, I meant that I ripped everything out that would allow the kernel image to be small enough to flash onto the phone. It does not boot however.

[Dmitry Vyukov]

Maybe this will help:

https://source.android.com/devices/tech/debug/kasan-kcov

I did not try to follow these instructions, though. But if we discover any additional issues, we can extend/correct the article.

--
You received this message because you are subscribed to the Google Groups "kasan-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kasa...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/215e1ec2-16b3-447d-9e76-3ef12853c957%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[MC]

Can you host your entire BoardConfig.mk somewhere? I have followed your instructions to the T, and when I flash the boot.img it just begins to bootloop. One difference however, is:

 BOARD_KERNEL_BASE        := 0x80000000
 BOARD_KERNEL_PAGESIZE    := 4096
-ifneq ($(filter marlin_kasan, $(TARGET_PRODUCT)),)
 BOARD_KERNEL_OFFSET      := 0x80000
 BOARD_KERNEL_TAGS_OFFSET := 0x02500000
 BOARD_RAMDISK_OFFSET     := 0x02700000
 BOARD_MKBOOTIMG_ARGS     := --kernel_offset $(BOARD_KERNEL_OFFSET) --ramdisk_offset $(BOARD_RAMDISK_OFFSET) --tags_offset $(BOARD_KERNEL_TAGS_OFFSET)
-else
-BOARD_KERNEL_TAGS_OFFSET := 0x02000000
-BOARD_RAMDISK_OFFSET     := 0x02200000 

-endif

My BoardConfig.mk does not have these lines that you have removed. It may be because I am building NJH47F, which is the August 2017 build, and not the absolute latest version.  I have in my BoardConfig.mk the following:

BOARD_KERNEL_BASE        := 0x80000000

BOARD_KERNEL_PAGESIZE    := 4096

BOARD_KERNEL_OFFSET      := 0x80000

BOARD_KERNEL_TAGS_OFFSET := 0x02500000

BOARD_RAMDISK_OFFSET     := 0x02700000

BOARD_MKBOOTIMG_ARGS     := --kernel_offset $(BOARD_KERNEL_OFFSET) --ramdisk_offset $(BOARD_RAMDISK_OFFSET) --tags_offset $(BOARD_KERNEL_TAGS_OFFSET)

TARGET_KERNEL_ARCH := arm64

TARGET_KERNEL_HEADER_ARCH := arm64

TARGET_KERNEL_CROSS_COMPILE_PREFIX := aarch64-linux-android-

TARGET_USES_UNCOMPRESSED_KERNEL := false

If you think I should just say screw it and move up to the absolute latest, I'll go ahead and give that a try but I'd prefer to stay on 7  

To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.

[MC]

Hi Dmitry, I realize I am pretty much talking to myself here, but hoping you will respond to this message or maybe this will help someone fumbling around on google.... I have nuked everything and started from scratch.  A couple things:

1. I had to use the latest android-ndk-16 beta version, NOT the latest stable (r15c or whatever) version. This was producing kernels that were huge and I was not able to flash them onto the device.

2. Speaking of the android-ndk, whenever I use that toolchain, I got a warning when building the kernel saying "KASAN: compiler doesn't support all options, trying a minimal configuration". Are you seeing this error too? If I download the Linaro gcc 6.x arm64 toolchain, I do not get this error message but then the kernel blows up to 30mb and it's too big to flash.

3. Even with the dramatically reduced kernel size using the latest NDK, when I configure with the marlin_defconfig and make only the changes you mentioned (remove LZ4, add slub debug, kcov, kasan), my kernel still is slightly too big (by about 2mb). I have to go in and remove some features in order for it to be small enough. Do you have to do this too? 

4. For anyone else having issues with bootloops: I was originally unable to boot freshly built AOSP images for Android 7.1.x (the device would go into bootloops), so I was forced to move up to Android 8.0. When I built the newer Oreo, this would not boot either to my dismay. I figured out that I had to flash a factory 8.0 image first, and then flash the 8.0 AOSP I had just built before. Not sure why, maybe this has something to do with some partition layout or something? Anyway, if anyone on Google is seeing this: first flash with an official factory image then flash the custom AOSP image next.

mike  

[Dmitry Vyukov]

On Sat, Oct 14, 2017 at 4:38 AM, MC <dar...@gmail.com> wrote:
>
> Hi Dmitry, I realize I am pretty much talking to myself here, but hoping you will respond to this message

Hi,

I never built andorid kernels for boards and know nothing about
android images, so I just don't have anything useful to say.

[bo Zhang]

Hi,

You can try this config file build.config.kasan in aosp oero source treee for msm. These config options are configed for building kernel with KASAN for pixel devices. It should be ok(I have succeeded).

This option CONFIG_CC_OPTIMIZE_FOR_SIZE is very important, it will significantly reduce the kernel image size.

Also, this article Dmitry provided is a good reference. https://source.android.com/devices/tech/debug/kasan-kcov

To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kasa...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/215e1ec2-16b3-447d-9e76-3ef12853c957%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "kasan-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kasan-dev/Ppz85NMGWLg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kasan-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kasa...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/CAF1V9aAC1XMfy3t14ArR0%3Dzr%2B%3Dz%2B5e6abPk_TF4aGyfAjNfFcw%40mail.gmail.com.

[MC]

Dude you're a lifesaver! CONFIG_CC_OPTIMIZE_FOR_SZE=y solved all of my problems!

To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.

To post to this group, send email to kasa...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/215e1ec2-16b3-447d-9e76-3ef12853c957%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "kasan-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kasan-dev/Ppz85NMGWLg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to kasan-dev+...@googlegroups.com.