Compiling 5.1.1 for a tablet + custom kernel sources

[Bim Overbohm]

Hi Group.

I have a Bay Trail Z3735F tablet (Medion Lifetab P8314.2 / MD99612) that has Android 5.0.2 as its last firmware version and kernel sources available. The support page can be found here.

I wanted to upgrade it and tried to compile Android-x86 5.1.1 from source on an Ubuntu 16.06.03 server VM. I first tried the Vanilla 5.1.1 x64 version and after fixing some errors can build an efi_ or iso_img and bootimage sucessfully. I documented my endeavors here. I used the vendors firmware package to flash the new firmware via fastboot by replacing the boot.img and system.img by my compiled versions (I had to img2simg the system.img to a sparse image first). I got some kernel output, but Android-x86 did not fully boot up... I'm not sure if this should have worked (at least with some missing functionality) or I have forgotten some steps or should have flashed some other images/partitions. I'm also unsure if the x86 or x64 version is correct for my device.

I figured I had the kernel sources, so I wanted to use them for a proper kernel, but I got no idea on how to integrate them into the build. The zip contains the folders "bootable", "external", "linux" and "vendor" and a "build.props" and a (kernel)".config". It even contains a shell script to compile, so I copied everything into my android-x86 folder. Running the shell script gives me a "bzImage" in a different output folder. I tried copying the bzImage to "out/target/product/x86_64/kernel", rebuild the boot.img and flash it, but only got a boot loop.
I also tried copying the kernel .config file to "kernel/arch/x86/configs/md99612.config" and running a regular android-x86_64 build using "m kernel TARGET_PRODUCT=android_x86_64 TARGET_KERNEL_CONFIG=md99612.config", but again got the same boot.img.

Help/Hint/Pointers on getting a running 5.1.1 would be great. Plus, if I get a running 5.1.1, how hard would it be to compile Android 7.1 or 8 for the device considering they should support more hardware and I have the kernel sources? Impossible or doable?

Best regards and thanks for helping out.

[lambdadroid]

Although I haven't looked at it in detail (someone else will be more qualified to answer this), as far as I know Android-x86 uses a modified boot procedure using GRUB and an extra ramdisk, which might not be directly applicable to the standard Android boot process that is likely used on your tablet. This might be why it doesn't boot.

The easiest way to see if Android-x86 would actually boot on your tablet would be to use the original ISO images and boot them from USB. At least on my Baytrail based tablet, booting from USB is possible using a powered OTG hub. That way I was able to boot right into the GUI and had the confirmation that there is hope to get something working ;) You might want to try that instead as a first step before trying to fit Android-x86 into the standard Android partition layout. Let me know if you want further information how USB booting works.

To answer the question regarding the original (stock) kernel: Two different approaches are possible if you want to run a newer Android version on your tablet, both have advantages and disadvantages:

- Use the ancient, outdated stock kernel:
    + Almost all hardware components should work out of the box.
    - Newer versions of Android-x86 are designed for recent upstream kernels (e.g. 4.14 or 4.9), you will run into issues with certain components where you'll either need to backport commits or make changes manually. This will only increase over time with newer Android versions...
    - A lot of unpatched bugs (especially security related)
- Use an upstream kernel:
    + Android-x86 (or similar) should work out of the box.
    + You get many fixes and improvements from upstream with little effort.
    - Some hardware components won't work out of the box: Depending on the hardware in the tablet, the two most common things I've seen are battery+charger drivers, as these are usually custom. Porting these from the stock kernel is not trivial and could take a significant amount of time. :/

For my tablet I chose the upstream kernel. It took quite a lot of time to get things working initially, but now I'm rather happy about it because I can just pull it upstream updates to get security fixes or support for newer Android versions.

[Bim Overbohm]

Am Samstag, 10. Februar 2018 16:04:44 UTC+1 schrieb lambdadroid:

Although I haven't looked at it in detail (someone else will be more qualified to answer this), as far as I know Android-x86 uses a modified boot procedure using GRUB and an extra ramdisk, which might not be directly applicable to the standard Android boot process that is likely used on your tablet. This might be why it doesn't boot.

The easiest way to see if Android-x86 would actually boot on your tablet would be to use the original ISO images and boot them from USB. At least on my Baytrail based tablet, booting from USB is possible using a powered OTG hub. That way I was able to boot right into the GUI and had the confirmation that there is hope to get something working ;) You might want to try that instead as a first step before trying to fit Android-x86 into the standard Android partition layout. Let me know if you want further information how USB booting works.

My tablet does not seem to boot from USB, even from a powered hub. I tried a couple of approaches. Flashing partitions works obviously, but else than that I could only boot from USB by starting into the tablets EFI shell and then running /EFI/boot/bootx64.efi or grubx64 efi. With the default Android-x86 5.1.1 kernel compiled into a boot.img the tablet at least seemed to boot to some extent, but seemingly failed to find the rootfs or partition needed later. Adding some insmod lines for a veriety of drivers (part_gpt, part_msdos etc.) did not seem to change anything. I guess I haven't understood the proper boot procedure for the tablet or lack some specific modules. Any pointer are welcome here...
 

To answer the question regarding the original (stock) kernel: Two different approaches are possible if you want to run a newer Android version on your tablet, both have advantages and disadvantages:

- Use the ancient, outdated stock kernel:
    + Almost all hardware components should work out of the box.
    - Newer versions of Android-x86 are designed for recent upstream kernels (e.g. 4.14 or 4.9), you will run into issues with certain components where you'll either need to backport commits or make changes manually. This will only increase over time with newer Android versions...
    - A lot of unpatched bugs (especially security related)
- Use an upstream kernel:
    + Android-x86 (or similar) should work out of the box.
    + You get many fixes and improvements from upstream with little effort.
    - Some hardware components won't work out of the box: Depending on the hardware in the tablet, the two most common things I've seen are battery+charger drivers, as these are usually custom. Porting these from the stock kernel is not trivial and could take a significant amount of time. :/

For my tablet I chose the upstream kernel. It took quite a lot of time to get things working initially, but now I'm rather happy about it because I can just pull it upstream updates to get security fixes or support for newer Android versions.

Ok, so an upstream kernel would be great as it might even have some of the required drivers and the rest could in theory be ported. Could you elaborate on how to use my kernel sources or any kernel sources in the android-x86 repo directory for building with that other kernel? Do I just dump the sources into the directory or what do I need to do?

[lambdadroid]

On Saturday, February 10, 2018 at 6:00:49 PM UTC+1, Bim Overbohm wrote: 

My tablet does not seem to boot from USB, even from a powered hub. I tried a couple of approaches. Flashing partitions works obviously, but else than that I could only boot from USB by starting into the tablets EFI shell and then running /EFI/boot/bootx64.efi or grubx64 efi. With the default Android-x86 5.1.1 kernel compiled into a boot.img the tablet at least seemed to boot to some extent, but seemingly failed to find the rootfs or partition needed later. Adding some insmod lines for a veriety of drivers (part_gpt, part_msdos etc.) did not seem to change anything. I guess I haven't understood the proper boot procedure for the tablet or lack some specific modules. Any pointer are welcome here...

For me it works like this:

  - Connect USB keyboard and your USB flash driver to the hub, and connect the external power to the hub

  - Turn on the tablet, and keep pressing F2 to enter the BIOS menu

  - Go to the boot menu. There is an additional option in the "Boot override" section, and I select that to boot from the USB flash drive.

However, if you can access the USB flash driver from the EFI shell that might be enough too: Which ISO of Android x86 did you try to run? Can you try the latest 64-bit release of Android Nougat (7.1) from http://www.android-x86.org/releases/releasenote-7-1-r1 ("android-x86_64-7.1-r1.iso"). I just verified that I can boot it successfully on my tablet.

(The rule of thumb is that if you use a recent upstream kernel, you can safely jump directly to a recent Android version. Older Android versions will only have an advantage for older kernel versions (e.g. your stock kernel).

 

Ok, so an upstream kernel would be great as it might even have some of the required drivers and the rest could in theory be ported. Could you elaborate on how to use my kernel sources or any kernel sources in the android-x86 repo directory for building with that other kernel? Do I just dump the sources into the directory or what do I need to do?

To be honest, I'm not sure where it is for Android-x86 5.x. There should be a "kernel.mk" file somewhere that points to the kernel sources and the kernel configuration. In general, you could probably try simply replacing them, and put the kernel configuration in the same place where it was before.

Note that I don't know much about Android-x86's build process because I've personally only used it for a few initial tests.

At some point, I set up a custom device configuration and now only re-use components of Android-x86/Android-IA in a custom ROM based on LineageOS. My experience is that it's easier to integrate into an existing Android device this way because you can keep the partition layout etc as-is, and don't need to change everything to employ Android-x86's boot setup (which is optimized for devices that weren't shipped with Android originally).

Plus, here an additional offer: For my device I have a port of the TWRP recovery based on an upstream kernel (Linux 4.14). If you're interested, I could try to remove a few device specific optimizations for my tablet and send you the recovery image. This should be pretty easy to test because you just need to flash the recovery partition and not system, data, cache etc. However, if you want to try this I'd need a few more information about your tablet first. A copy of the kernel log (using dmesg) on the stock ROM/recovery and the partition layout would be a good start.

[Bim Overbohm]

I can not enter the BIOS on the tablet. Theres no reaction to F1 / F2 / Del and so on when booting. The only way to enter the BIOS is to run the efi shell via POWER + VOLUME UP, run grubx64.efi and select the option to start the BIOS. This works, but leaves me with a password input field, so I can't enter the BIOS and change any options there. I tried reflashing the BIOS with AfuEfix64, erasing the NVRAM etc., but the password protection does not go away. I also tried different default passwords, but to no avail. I have now contacted the vendor to get the password. Fingers crossed.

Running Android 7.1 Live from USB shows these messages (postimg link) and then it just sits there... The USB drive and keyboard do not seem to be powered anymore, so I guess it is a USB driver problem. When I had flashed the boot.img and system.img the tablet got further in the boot process. Maybe then the partition layout was'nt correct for Android to boot up though. Not sure.

Thanks for the offer with TWRP. It is tempting, but I'm a bit anxious about trashing the current recovery and possibly fastboot, because it's my only option to unbrick the device atm... :/

[lambdadroid]

On Sunday, February 11, 2018 at 12:45:29 AM UTC+1, Bim Overbohm wrote:

I can not enter the BIOS on the tablet. Theres no reaction to F1 / F2 / Del and so on when booting. The only way to enter the BIOS is to run the efi shell via POWER + VOLUME UP, run grubx64.efi and select the option to start the BIOS. This works, but leaves me with a password input field, so I can't enter the BIOS and change any options there. I tried reflashing the BIOS with AfuEfix64, erasing the NVRAM etc., but the password protection does not go away. I also tried different default passwords, but to no avail. I have now contacted the vendor to get the password. Fingers crossed.

Running Android 7.1 Live from USB shows these messages (postimg link) and then it just sits there... The USB drive and keyboard do not seem to be powered anymore, so I guess it is a USB driver problem. When I had flashed the boot.img and system.img the tablet got further in the boot process. Maybe then the partition layout was'nt correct for Android to boot up though. Not sure.

Although your image link is broken, I'm not sure there is much we can do regarding USB boot. I don't have any further suggestions for that right now.

 

Thanks for the offer with TWRP. It is tempting, but I'm a bit anxious about trashing the current recovery and possibly fastboot, because it's my only option to unbrick the device atm... :/

Note that if you play with stuff like this, you must always be prepared that you might brick the device eventually. It's important to be careful and to check possible recovery options before you start, but you can never know what happens. :)

I personally have *never* touched the Fastboot partition on my device, so I always had it as a fallback to flash new boot and recovery partitions. Sooner or later you'll want to replace the recovery anyway, so if you can flash the recovery partition from Fastboot you should be fine.

HOWEVER, as a first step there might be still another option: If your boot partition is big enough, you can technically also just flash the recovery image to the boot partition. I've done this before and it works fine because the recovery partition is just a regular boot partition otherwise. That way you could keep the recovery partition as-is for now.

[Bim Overbohm]

Posting the link again: https://s10.postimg.org/bxam09icp/IMG_20180210_215820.jpg

The orignal firmware said Kernel 3.10.something, the kernel sources say that too. Will try to find out where to put the kernel sources and build an iso again. I hope I get the BIOS password...

When I got proper dmesg output I'll send you a link for a recovery. Is the fastboot partition the droidboot.img or what name does it go by usually?
Here ist the current partition layout of the device btw:

partition_table=gpt
create -z /dev/block/mmcblk0
create /dev/block/mmcblk0
boot -p /dev/block/mmcblk0
reload /dev/block/mmcblk0
add -b 40 -s 131072 -t efi -u C12A7328-F81F-11D2-BA4B-00A0C93EC93B -l ESP -T 0 -P 0 /dev/block/mmcblk0
add -b 131112 -s 131072 -t data -u 80868086-8086-8086-8086-FFFFFFFFFFF0 -l reserved -T 0 -P 0 /dev/block/mmcblk0
add -b 262184 -s 32768 -t data -u 80868086-8086-8086-8086-000000000100 -l boot -T 3 -P 15 /dev/block/mmcblk0
add -b 294952 -s 32768 -t data -u 80868086-8086-8086-8086-000000000101 -l recovery -T 3 -P 14 /dev/block/mmcblk0
add -b 327720 -s 32768 -t data -u 80868086-8086-8086-8086-000000000102 -l fastboot -T 3 -P 13 /dev/block/mmcblk0
add -b 360488 -s 32768 -t data -u 80868086-8086-8086-8086-FFFFFFFFFFF1 -l reserved_1 -T 3 -P 0 /dev/block/mmcblk0
add -b 393256 -s 131072 -t data -u 80868086-8086-8086-8086-000000000104 -l test -T 3 -P 12 /dev/block/mmcblk0
add -b 524328 -s 65536 -t data -u 80868086-8086-8086-8086-000000000001 -l panic -T 0 -P 0 /dev/block/mmcblk0
add -b 589864 -s 32768 -t data -u 80868086-8086-8086-8086-000000000002 -l factory -T 0 -P 0 /dev/block/mmcblk0
add -b 622632 -s 32768 -t data -u 80868086-8086-8086-8086-000000000003 -l misc -T 0 -P 0 /dev/block/mmcblk0
add -b 655400 -s 32768 -t data -u 80868086-8086-8086-8086-000000000004 -l config -T 0 -P 0 /dev/block/mmcblk0
add -b 688168 -s 2097152 -t data -u 80868086-8086-8086-8086-000000000005 -l cache -T 0 -P 0 /dev/block/mmcblk0
add -b 2785320 -s 65536 -t data -u 80868086-8086-8086-8086-000000000006 -l logs -T 0 -P 0 /dev/block/mmcblk0
add -b 2850856 -s 4194304 -t data -u 80868086-8086-8086-8086-000000000007 -l system -T 0 -P 0 /dev/block/mmcblk0
add -b 7045160 -s $calc($lba_end-33) -t data -u 80868086-8086-8086-8086-000000000008 -l data -T 0 -P 0 /dev/block/mmcblk0
reload /dev/block/mmcblk0

[lambdadroid]

On Sunday, February 11, 2018 at 12:26:10 PM UTC+1, Bim Overbohm wrote:

Posting the link again: https://s10.postimg.org/bxam09icp/IMG_20180210_215820.jpg

The orignal firmware said Kernel 3.10.something, the kernel sources say that too. Will try to find out where to put the kernel sources and build an iso again. I hope I get the BIOS password...

Yes, those devices were shipped with ancient Linux 3.10 kernels. They have been EOL for quite a while already... They did make quite a lot of changes to the kernel back then, but over time, a lot of the needed changes also arrived upstream. For some components however, they decided to hardcode things in the kernel instead of setting up the BIOS/ACPI/etc properly. I had to do minor fixes for most components to get them working. The fixes are simple, but finding the actual problems took a lot of time.

I've never even built my stock kernel from source because they didn't share their kernel config. I've never really bothered with it and started testing with upstream kernels directly. After quite a bit of work it works "well enough" now, but obviously I cannot say if you might have more luck with the stock kernel. I can't help you much with that one, though.

 

When I got proper dmesg output I'll send you a link for a recovery. Is the fastboot partition the droidboot.img or what name does it go by usually?

I'm not sure if it's the same on your device, but on my device there are two Fastboot implementations. One is called "Droidboot" and this is the usual fastboot interface where you can flash boot, recovery, pretty much all partitions. Similar to the recovery partition, it's loaded by the Android bootloader, comes with a regular (3.10, urgh) Linux kernel etc.

There is another Fastboot implementation though, which is much earlier in the boot process. It's called "DNX mode" and it's supposed to be used for recovery if even the bootloader and Fastboot partition were destroyed. As far as I know, it's directly built into the BIOS of the device, because it boots even before the BIOS menu shows up when pressing F2. On my device I can enter DNX mode by pressing Volume Up + Volume Down + Power when turning on the tablet. It then only shows 3 lines "Fastboot starting...".