[llvm-dev] Linking Linux kernel with LLD

[Dmitry Golovin via llvm-dev]

Hi all!

Recently there was a discussion on this mailing list about linking Linux and GRUB with LLD.

I have actually tried to link Linux kernel with LLD and (with some modifications) it linked successfully (but didn't run well).

Here is the list of modifications I had to do in order to link the kernel (I have used llvmlinux with clang and mainline with gcc, the results are similar):

1. LLD patches:
- D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
- D28857 (Allow relative relocations to a absolute value defined in linker script)
- [not actually used] D28612 (Added support for --emit-relocs)

2. Kernel configuration. I tried "make tinyconfig" and then changed to CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration for x86_64. After I get it working, I will try defconfig.

3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think that cc-option should test if compiler supports '-m16' option and if it does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h' instead. I don't know why, but this test failed for me and LLD didn't accept this header, so I just simply changed the flag to '-m16' because I didn't want to investigate why did the test failed. It's just a quick workaround, but it works.

3. In arch/x86/boot/Makefile and arch/x86/realmode/rm/Makefile added "-m elf_i386" to LDFLAGS_setup.elf and LDFLAGS_realmode.elf accordingly. The problem is that the linker is invoked as "ld.lld -m elf_x86_64" and it ignores the OUTPUT_FORMAT from linker scripts. Being invoked as "ld.lld -m elf_x86_64 -m elf_i386" doesn't break anything and LLD even produces the right format of the binaries. Again a quick workaround, but it works.

4. In arch/x86/realmode/rm/Makefile removed "--emit-relocs" from LDFLAGS_realmode.elf. This is the problem with D28612 — I applied the patch to support --emit-relocs option, but LLD segfaulted. Reported here: https://llvm.org/bugs/show_bug.cgi?id=31579#c8

5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", because LLD doesn't support it.

6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of 2560 because it doesn't seem that LLD supports math inside ASSERT in linker scripts.

7. In arch/x86/boot/setup.ld commented out the line ". = ASSERT(_end <= 0x8000, "Setup too big!");" because it seems to always fire this assertion, and the kernel is not going to be working anyway, so I just want it linked even if it will be unusable.

8. In arch/x86/lib/copy_user_64.S removed the line "bad_from_user:" (probably it was just a mistake), in arch/x86/kernel/vmlinux.lds.S removed semicolon after "*(.apicdrivers)" (also probably just a mistake).

Finally the kernel was built and it obviously didn't run (only printed out "No setup signature found..." but this is some result as well). Probably, the result could be better if the --emit-relocs option didn't fail and CONSTRUCTORS were supported. I really don't know what to do about the assertion that I have commented out.

I also tried using tools provided by LLVM instead of binutils (HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm), it seems to work with llvmlinux, but clang will not work with the mainline. Clang doesn't seem to use integrated assembler, so I had to write a simple wrapper script around llvm-mc to use instead GNU as. I also have few patches to make llvm-objcopy to work with latest LLVM and support needed subset of flags, will publish it soon.

You can find my workarounds as a patch (the kernel will not work with this applied!) and my kernel config attached.

Regards,
Dmitry

[Rui Ueyama via llvm-dev]

Wow, great work! I'm sure you are the first person who linked the Linux kernel using LLD. :)

I think "no setup signature found" error is an error of the boot loader (probably grub). The resulting binary probably lacked magic bytes or something that a boot loader uses to identify bootable images. It should be fixable by investigating what is actually missing and fix it.

On Thu, Jan 19, 2017 at 1:56 PM, Dmitry Golovin via llvm-dev <llvm...@lists.llvm.org> wrote:

Hi all!

Recently there was a discussion on this mailing list about linking Linux and GRUB with LLD.

I have actually tried to link Linux kernel with LLD and (with some modifications) it linked successfully (but didn't run well).

Here is the list of modifications I had to do in order to link the kernel (I have used llvmlinux with clang and mainline with gcc, the results are similar):

1. LLD patches:
  - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
  - D28857 (Allow relative relocations to a absolute value defined in linker script)
  - [not actually used] D28612 (Added support for --emit-relocs)

2. Kernel configuration. I tried "make tinyconfig" and then changed to CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration for x86_64. After I get it working, I will try defconfig.

3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think that cc-option should test if compiler supports '-m16' option and if it does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h' instead. I don't know why, but this test failed for me and LLD didn't accept this header, so I just simply changed the flag to '-m16' because I didn't want to investigate why did the test failed. It's just a quick workaround, but it works.

3. In arch/x86/boot/Makefile and arch/x86/realmode/rm/Makefile added "-m elf_i386" to LDFLAGS_setup.elf and LDFLAGS_realmode.elf accordingly. The problem is that the linker is invoked as "ld.lld -m elf_x86_64" and it ignores the OUTPUT_FORMAT from linker scripts. Being invoked as "ld.lld -m elf_x86_64 -m elf_i386" doesn't break anything and LLD even produces the right format of the binaries. Again a quick workaround, but it works.

4. In arch/x86/realmode/rm/Makefile removed "--emit-relocs" from LDFLAGS_realmode.elf. This is the problem with D28612 — I applied the patch to support --emit-relocs option, but LLD segfaulted. Reported here: https://llvm.org/bugs/show_bug.cgi?id=31579#c8

5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", because LLD doesn't support it.

We need to support those missing features.

 

6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of 2560 because it doesn't seem that LLD supports math inside ASSERT in linker scripts.

So this is the line.

  . = ASSERT(__end_init <= 5*512, "init sections too big!");

and LLD should be able to handle this pattern already, so I think you found a bug. We need to fix that. (The code to handle ASSERT is here, so if you are interested, you can take a look. It is a simple recursive-descendent parser, and because readAssert calls readExpr (as opposed to next() which returns the next token), it is intended to be able to handle an expression inside ASSERT.)

 

7. In arch/x86/boot/setup.ld commented out the line ". = ASSERT(_end <= 0x8000, "Setup too big!");" because it seems to always fire this assertion, and the kernel is not going to be working anyway, so I just want it linked even if it will be unusable.

That's a weird issue... need to take a look deeper.

 

8. In arch/x86/lib/copy_user_64.S removed the line "bad_from_user:" (probably it was just a mistake), in arch/x86/kernel/vmlinux.lds.S removed semicolon after "*(.apicdrivers)" (also probably just a mistake).

Finally the kernel was built and it obviously didn't run (only printed out "No setup signature found..." but this is some result as well). Probably, the result could be better if the --emit-relocs option didn't fail and CONSTRUCTORS were supported. I really don't know what to do about the assertion that I have commented out.

I also tried using tools provided by LLVM instead of binutils (HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm), it seems to work with llvmlinux, but clang will not work with the mainline. Clang doesn't seem to use integrated assembler, so I had to write a simple wrapper script around llvm-mc to use instead GNU as. I also have few patches to make llvm-objcopy to work with latest LLVM and support needed subset of flags, will publish it soon.

You can find my workarounds as a patch (the kernel will not work with this applied!) and my kernel config attached.

Regards,
Dmitry

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[George Rimar via llvm-dev]

Hi Dmitry,

thanls for sharing. Few comments/questions below:

>Here is the list of modifications I had to do in order to link the kernel (I have used llvmlinux with clang and mainline with gcc, the >results are similar):
>
>1. LLD patches:
> - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)

Do you remember where it was used ?

>5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", because LLD doesn't support it.

It is https://reviews.llvm.org/D28951. CONSTRUCTORS can be just removed, they do nothing for ELF.

>6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of 2560 because it doesn't seem that LLD supports math inside >ASSERT in linker scripts.

It is actually not relative with ASSERT. LLD does not support "symbol = 5*6", but accepts "symbol = 5 * 6" currently.
Not sure what is easy fix here.

>Finally the kernel was built and it obviously didn't run (only printed out "No setup signature found..." but this is some result as well). >Probably, the result could be better if the --emit-relocs option didn't fail and CONSTRUCTORS were supported. I really don't know what >to do about the assertion that I have commented out.

I updated patch for --emit-relocs, now they do not fail: https://reviews.llvm.org/D28612
It looks to be important feature for self relocations, so it is not surprising it did not run without :)

George.

[Sean Silva via llvm-dev]

On Fri, Jan 20, 2017 at 8:35 AM, George Rimar via llvm-dev <llvm...@lists.llvm.org> wrote:

Hi Dmitry,

thanls for sharing. Few comments/questions below:

>Here is the list of modifications I had to do in order to link the kernel (I have used llvmlinux with clang and mainline with gcc, the >results are similar):
>
>1. LLD patches:
 > - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)

Do you remember where it was used ?

>5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", because LLD doesn't support it.

It is https://reviews.llvm.org/D28951. CONSTRUCTORS can be just removed, they do nothing for ELF.

>6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of 2560 because it doesn't seem that LLD supports math inside >ASSERT in linker scripts.

It is actually not relative with ASSERT. LLD does not support "symbol = 5*6", but accepts "symbol = 5 * 6" currently.
Not sure what is easy fix here.

I'm not sure if it is easy, but I think that it's clear that the linkerscript lexer needs to be improved. I think that is the source of the problems with `*(.apicdrivers);` as well. This is not the first bug related to lexing that we have run into (e.g. lexing `.=` as a single token is the cause of https://llvm.org/bugs/show_bug.cgi?id=31128 ).

-- Sean Silva

[Dmitry Golovin via llvm-dev]

>> - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
> Do you remember where it was used ?

I can undo the patch (but I can access the build machine only on Monday) and see what breaks.

> CONSTRUCTORS can be just removed, they do nothing for ELF.

Okay, this is what I did (I thought it will break things, but it is okay). I will apply the patch.

> LLD does not support "symbol = 5*6", but accepts "symbol = 5 * 6" currently. Not sure what is easy fix here.

Just ignore it for now, it's not really a big deal.

> I updated patch for --emit-relocs, now they do not fail: https://reviews.llvm.org/D28612

Thank you, I will apply the updated patch and hope that it will boot.

Regards,
Dmitry

20.01.2017, 18:36, "George Rimar" <gri...@accesssoftek.com>:

[Sean Silva via llvm-dev]

On Fri, Jan 20, 2017 at 12:44 PM, Sean Silva <chiso...@gmail.com> wrote:

On Fri, Jan 20, 2017 at 8:35 AM, George Rimar via llvm-dev <llvm...@lists.llvm.org> wrote:

Hi Dmitry,

thanls for sharing. Few comments/questions below:

>Here is the list of modifications I had to do in order to link the kernel (I have used llvmlinux with clang and mainline with gcc, the >results are similar):
>
>1. LLD patches:
 > - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)

Do you remember where it was used ?

>5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", because LLD doesn't support it.

It is https://reviews.llvm.org/D28951. CONSTRUCTORS can be just removed, they do nothing for ELF.

>6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of 2560 because it doesn't seem that LLD supports math inside >ASSERT in linker scripts.

It is actually not relative with ASSERT. LLD does not support "symbol = 5*6", but accepts "symbol = 5 * 6" currently.
Not sure what is easy fix here.

I'm not sure if it is easy, but I think that it's clear that the linkerscript lexer needs to be improved. I think that is the source of the problems with `*(.apicdrivers);` as well.

Actually, quickly staring at the code, the `*(.apicdrivers);` seems like it will be lexed  correctly.

-- Sean Silva

[Sean Silva via llvm-dev]

On Thu, Jan 19, 2017 at 1:56 PM, Dmitry Golovin via llvm-dev <llvm...@lists.llvm.org> wrote:

Hi all!

Recently there was a discussion on this mailing list about linking Linux and GRUB with LLD.

I have actually tried to link Linux kernel with LLD and (with some modifications) it linked successfully (but didn't run well).

Awesome! Thanks for digging into this!

When Michael, Davide, and I were debugging the issues with running the LLD-linked FreeBSD kernel, we found that there was really no alternative but to get familiar with the bootloader code and do printf debugging to understand what the bootloader didn't like. (even hooking into qemu's debugger integration didn't work; it is just too early for the usual debugger integration to work I think (interrupts and stuff not in a "sane" state etc.)). I'm happy to share more info about that experience if it would be useful to you.

I'm guessing Ed (CC'd) also had some similar experiences getting the FreeBSD bootloaders working with LLD and might have some advice as well w.r.t. a good workflow and what tools to use.

 

Here is the list of modifications I had to do in order to link the kernel (I have used llvmlinux with clang and mainline with gcc, the results are similar):

1. LLD patches:
  - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
  - D28857 (Allow relative relocations to a absolute value defined in linker script)
  - [not actually used] D28612 (Added support for --emit-relocs)

2. Kernel configuration. I tried "make tinyconfig" and then changed to CONFIG_64BIT=y with menuconfig. I wanted the minimal kernel configuration for x86_64. After I get it working, I will try defconfig.

3. In arch/x86/Makefile changed the M16_CFLAGS to just "-m16". I think that cc-option should test if compiler supports '-m16' option and if it does, use it, if it doesn't use '-m32 -Wa,./arch/x86/boot/code16gcc.h' instead. I don't know why, but this test failed for me and LLD didn't accept this header, so I just simply changed the flag to '-m16' because I didn't want to investigate why did the test failed. It's just a quick workaround, but it works.

These seem to be compiler flags. It is probably best to use GCC as the compiler for now (i.e. only swap out the linker). That will allow focusing on the LLD issues. Clang (and other LLVM tools) have their own issues with GNU compatibility, but things will be more difficult to debug when there are more moving pieces.

-- Sean Silva

 

3. In arch/x86/boot/Makefile and arch/x86/realmode/rm/Makefile added "-m elf_i386" to LDFLAGS_setup.elf and LDFLAGS_realmode.elf accordingly. The problem is that the linker is invoked as "ld.lld -m elf_x86_64" and it ignores the OUTPUT_FORMAT from linker scripts. Being invoked as "ld.lld -m elf_x86_64 -m elf_i386" doesn't break anything and LLD even produces the right format of the binaries. Again a quick workaround, but it works.

4. In arch/x86/realmode/rm/Makefile removed "--emit-relocs" from LDFLAGS_realmode.elf. This is the problem with D28612 — I applied the patch to support --emit-relocs option, but LLD segfaulted. Reported here: https://llvm.org/bugs/show_bug.cgi?id=31579#c8

5. In arch/x86/kernel/vmlinux.lds.S commented out the "CONSTRUCTORS", because LLD doesn't support it.

6. In arch/x86/boot/setup.ld replaced 5*512 with precalculated value of 2560 because it doesn't seem that LLD supports math inside ASSERT in linker scripts.

7. In arch/x86/boot/setup.ld commented out the line ". = ASSERT(_end <= 0x8000, "Setup too big!");" because it seems to always fire this assertion, and the kernel is not going to be working anyway, so I just want it linked even if it will be unusable.

8. In arch/x86/lib/copy_user_64.S removed the line "bad_from_user:" (probably it was just a mistake), in arch/x86/kernel/vmlinux.lds.S removed semicolon after "*(.apicdrivers)" (also probably just a mistake).

Finally the kernel was built and it obviously didn't run (only printed out "No setup signature found..." but this is some result as well). Probably, the result could be better if the --emit-relocs option didn't fail and CONSTRUCTORS were supported. I really don't know what to do about the assertion that I have commented out.

I also tried using tools provided by LLVM instead of binutils (HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm), it seems to work with llvmlinux, but clang will not work with the mainline. Clang doesn't seem to use integrated assembler, so I had to write a simple wrapper script around llvm-mc to use instead GNU as. I also have few patches to make llvm-objcopy to work with latest LLVM and support needed subset of flags, will publish it soon.

You can find my workarounds as a patch (the kernel will not work with this applied!) and my kernel config attached.

Regards,
Dmitry

[Sean Silva via llvm-dev]

One thing I remember is that Linux's build system does directly tamper with object files with custom tools, so those tools might be unhappy with LLD's binaries or just do the wrong thing (sometimes such tools have hardcoded assumptions etc.). One specific case I remember is that linux's copy_from_user/copy_to_user implementations use inline asm to build up a table which is sorted by a custom tool (I think this modification happens post-link even).

http://lxr.free-electrons.com/source/arch/x86/include/asm/uaccess.h

-- Sean Silva

On Thu, Jan 19, 2017 at 1:56 PM, Dmitry Golovin via llvm-dev <llvm...@lists.llvm.org> wrote:

[Dmitry Golovin via llvm-dev]

>> - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
> Do you remember where it was used ?

setup.elf:
ld.lld -m elf_i386 -T arch/x86/boot/setup.ld arch/x86/boot/a20.o arch/x86/boot/bioscall.o arch/x86/boot/cmdline.o arch/x86/boot/copy.o arch/x86/boot/cpu.o arch/x86/boot/cpuflags.o arch/x86/boot/cpucheck.o arch/x86/boot/early_serial_console.o arch/x86/boot/edd.o arch/x86/boot/header.o arch/x86/boot/main.o arch/x86/boot/mca.o arch/x86/boot/memory.o arch/x86/boot/pm.o arch/x86/boot/pmjump.o arch/x86/boot/printf.o arch/x86/boot/regs.o arch/x86/boot/string.o arch/x86/boot/tty.o arch/x86/boot/video.o arch/x86/boot/video-mode.o arch/x86/boot/version.o arch/x86/boot/video-vga.o arch/x86/boot/video-vesa.o arch/x86/boot/video-bios.o -o arch/x86/boot/setup.elf
ld.lld: error: do not know how to handle relocation 'R_386_PC8' (23)

> I updated patch for --emit-relocs, now they do not fail.

Thanks, applied it, doesn't fail.

I still didn't do anything with "Setup too big!" problem, just commented out the assert. Tried booting the resulting bzImage and vmlinux with qemu. The bzImage only did reboot over and over, but the vmlinux did show an adorable picture (attached).

Regards,
Dmitry

20.01.2017, 22:49, "Dmitry Golovin via llvm-dev" <llvm...@lists.llvm.org>:

[George Rimar via llvm-dev]

>I'm not sure if it is easy, but I think that it's clear that the linkerscript lexer needs to be improved. I think that is the source of the >problems with `*(.apicdrivers);` as well. This is not the first bug related to lexing that we have run into (e.g. lexing `.=` as a single >token is the cause of https://llvm.org/bugs/show_bug.cgi?id=31128 ).

>

>-- Sean Silva

 

PR31128​ seems to be not an issue. Both gold and bfd do not accept '.='.

So it seems the only known issue we have is about math expressions like "x = 5*4",

I am going to look again how to fix that.

George.​

[Rui Ueyama via llvm-dev]

Our tokenizer recognize

  [A-Za-z0-9_.$/\\~=+[]*?\-:!<>]+

as a token. gold uses more complex rules to tokenize. I don't think we need that much complex rules, but there seems to be room to improve our tokenizer. In particular, I believe we can parse the Linux's linker script by changing the tokenizer rules as follows.

  [A-Za-z_.$/\\~=+[]*?\-:!<>][A-Za-z0-9_.$/\\~=+[]*?\-:!<>]*

or

  [0-9]+

[Sean Silva via llvm-dev]

On Mon, Jan 23, 2017 at 8:16 AM, Dmitry Golovin via llvm-dev <llvm...@lists.llvm.org> wrote:

>>  - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
> Do you remember where it was used ?

setup.elf:
      ld.lld -m elf_i386 -T arch/x86/boot/setup.ld arch/x86/boot/a20.o arch/x86/boot/bioscall.o arch/x86/boot/cmdline.o arch/x86/boot/copy.o arch/x86/boot/cpu.o arch/x86/boot/cpuflags.o arch/x86/boot/cpucheck.o arch/x86/boot/early_serial_console.o arch/x86/boot/edd.o arch/x86/boot/header.o arch/x86/boot/main.o arch/x86/boot/mca.o arch/x86/boot/memory.o arch/x86/boot/pm.o arch/x86/boot/pmjump.o arch/x86/boot/printf.o arch/x86/boot/regs.o arch/x86/boot/string.o arch/x86/boot/tty.o arch/x86/boot/video.o arch/x86/boot/video-mode.o arch/x86/boot/version.o arch/x86/boot/video-vga.o arch/x86/boot/video-vesa.o arch/x86/boot/video-bios.o -o arch/x86/boot/setup.elf
    ld.lld: error: do not know how to handle relocation 'R_386_PC8' (23)

> I updated patch for --emit-relocs, now they do not fail.

Thanks, applied it, doesn't fail.

I still didn't do anything with "Setup too big!" problem, just commented out the assert. Tried booting the resulting bzImage and vmlinux with qemu. The bzImage only did reboot over and over, but the vmlinux did show an adorable picture (attached).

That's beautiful! Looks like some kernel or loader text got copied into the VGA text buffer. Those "e with two dots above" looks like Code Page 437 (https://en.wikipedia.org/wiki/Code_page_437) for 0x89 which is a common MOV opcode in x86 machine code. "Capital H" and "capital Phi" also look like common x86 opcodes.

I remember when we were getting the FreeBSD kernel to link correctly, one of the bugs was that the kernel load address was too low (because of our MAXPAGESIZE value) which meant that the location where the kernel was asking to be copied into actually overlapped the bootloader's text, which ended about as well as you would expect it to. We didn't get a pretty picture though :)

-- Sean Silva

[George Rimar via llvm-dev]

>>> - D28094 (Implemented support for R_386_PC8/R_386_8 relocations)
>> Do you remember where it was used ?
>
>setup.elf:

> ld.lld -m elf_i386 -T arch/x86/boot/setup.ld arch/x86/boot/a20.o arch/x86/boot/bioscall.o arch/x86/boot/cmdline.o arch/x86/boot/copy.o arch/x86/boot/cpu.o >arch/x86/boot/cpuflags.o arch/x86/boot/cpucheck.o arch/x86/boot/early_serial_console.o arch/x86/boot/edd.o arch/x86/boot/header.o arch/x86/boot/main.o >arch/x86/boot/mca.o arch/x86/boot/memory.o arch/x86/boot/pm.o arch/x86/boot/pmjump.o arch/x86/boot/printf.o arch/x86/boot/regs.o arch/x86/boot/string.o >arch/x86/boot/tty.o arch/x86/boot/video.o arch/x86/boot/video-mode.o arch/x86/boot/version.o arch/x86/boot/video-vga.o arch/x86/boot/video-vesa.o arch/x86/boot/video->bios.o -o arch/x86/boot/setup.elf

> ld.lld: error: do not know how to handle relocation 'R_386_PC8' (23)

Thanks, I'll take a look and probably resurrect that abandoned patch then.

>I still didn't do anything with "Setup too big!" problem, just commented out the assert.

I did not get to that step yet, but just as a guess: you probably can try to add -N option for that place.

>Tried booting the resulting bzImage and vmlinux with qemu. The bzImage only did reboot over and over, but the vmlinux did show an adorable picture (attached).
>

That is interesting information, thanks.
I am not familar with linux kernel, but testing with qemu helped a lot when we worked on booting freebsd loaders.
I hope to get to that step soon, after finishing with other already known issues.

[George Rimar via llvm-dev]

>Our tokenizer recognize

>

>  [A-Za-z0-9_.$/\\~=+[]*?\-:!<>]+

>

>as a token. gold uses more complex rules to tokenize. I don't think we need that much complex rules, but there seems to be >room to improve our tokenizer. In particular, I believe we can parse the Linux's linker script by changing the tokenizer rules as >follows.

>

>  [A-Za-z_.$/\\~=+[]*?\-:!<>][A-Za-z0-9_.$/\\~=+[]*?\-:!<>]*

>

>or

>

>  [0-9]+

That probably should help a bit,

but does not solve a problem in general.

I think it will not work for expressions like

. = z5*4;

as it will read "z5*4" as a single token I think.

I was thinking about entering some special parser state for 

extracting sub tokens from tokens transparently when

we are inside code that evaluates the expression.

We can start from your suggestion first I think and see how it works

and if we really face scripts writtel like above in real life.

At least it is not harmfull and should help to kernel.

I'll try to prepare a patch if you do not mind.

George.

[George Rimar via llvm-dev]

>Our tokenizer recognize

>

>  [A-Za-z0-9_.$/\\~=+[]*?\-:!<>]+

>

>as a token. gold uses more complex rules to tokenize. I don't think we need that much complex rules, but there seems to be >room to improve our tokenizer. In particular, I believe we can parse the Linux's linker script by changing the tokenizer rules as >follows.

>

>  [A-Za-z_.$/\\~=+[]*?\-:!<>][A-Za-z0-9_.$/\\~=+[]*?\-:!<>]*

>

>or

>

>  [0-9]+​

After more investigation, that seems will not work so simple.

Next are possible examples where it will be broken:

. = 0x1000; (gives tokens "0, x1000")

. = A*10;   (gives "A*10")

. = 10k;    (gives "10, k")

. = 10*5;   (gives "10, *5"

"[0-9]+" could be "[0-9][kmhKMHx0-9]*"

but for "10*5" that anyways gives "10" and "*5" tokens.

And I do not think we can involve some handling of operators,

as its hard to assume some context on tokenizing step.

We do not know if that a file name we are parsing or a math expression.

May be worth trying to handle this on higher level, during evaluation of

expressions ?

George.

[Rui Ueyama via llvm-dev]

Well, maybe, we should just change the Linux kernel instead of tweaking our tokenizer too hard.

[Dmitry Golovin via llvm-dev]

Thanks for merging the patches into upstream. Now there are only two left: D28094 and D28612.

> just as a guess: you probably can try to add -N option for that place

The "-N" option (whatever it is) doesn't help. I can compare the object files produced by bfd, gold and lld.

I have attached my kernel config and kernel patch for upstream version. You can also examine my vmlinux and run it with

qemu-system-x86_64 -kernel linux -append 'console=ttyS0' -serial stdio

Regards,
Dmitry

24.01.2017, 10:47, "George Rimar" <gri...@accesssoftek.com>:

[George Rimar via llvm-dev]

> Well, maybe, we should just change the Linux kernel instead of tweaking our tokenizer too hard.

I agree, for now I am inclined to do that and watch for other scripts.

George.

[George Rimar via llvm-dev]

Hi !

I am observing next issue when linking with WIP patches applied:

ld: warning: cannot find entry symbol _start; defaulting to 0x1000
RELOCS arch/x86/realmode/rm/realmode.relocs
OBJCOPY arch/x86/realmode/rm/realmode.bin
objcopy:arch/x86/realmode/rm/realmode.elf: Bad value
arch/x86/realmode/rm/Makefile:61: recipe for target 'arch/x86/realmode/rm/realmode.bin' failed
make[5]: *** [arch/x86/realmode/rm/realmode.bin] Error 1

It seems to be objcopy version specific.

With GNU objcopy (GNU Binutils for Ubuntu) 2.26.1 and GNU objcopy (GNU Binutils) 2.27 I have the same errors,
GNU objcopy (GNU Binutils) 2.17.50.20070806 produces output fine.

What version of binutils do you use ?

Also I am observing that we do not produce correct output for realmode.elf anyways.
It has next script:
(arch/x86/realmode/rm/realmode.lds.S)

.rodata : {
*(.rodata)
*(.rodata.*)
. = ALIGN(16);
video_cards = .;
*(.videocards)
video_cards_end = .;
}

But symbols video_cards == video_cards_end in LLD output (realmode.elf),
what does not seems to be correct.

So I believe realmode.elf, bin are broken, even if objcopy runs fine.

I am investigating it now.

[Dmitry Golovin via llvm-dev]

Hi!

I used objcopy from GNU Binutils 2.27 and I didn't have this error. Maybe it's something new, so I will build a new version of LLD and try to reproduce.

Sorry, I accidentally used "Reply" instead of "Reply all", so the message didn't get to the mailing list.

Regards,
Dmitry

25.01.2017, 18:19, "George Rimar" <gri...@accesssoftek.com>:

[Sean Silva via llvm-dev]

On Tue, Jan 24, 2017 at 11:29 AM, Rui Ueyama <ru...@google.com> wrote:

Well, maybe, we should just change the Linux kernel instead of tweaking our tokenizer too hard.

This is silly. Writing a simple and maintainable lexer is not hard (look e.g. at https://reviews.llvm.org/D10817). There are some complicated context-sensitive cases in linker scripts that break our approach of tokenizing up front (so we might want to hold off on), but we aren't going to die from implementing enough to lex basic arithmetic expressions independent of whitespace.

We will be laughed at. ("You seriously couldn't even be bothered to implement a real lexer?")

-- Sean Silva

[Sean Silva via llvm-dev]

On Thu, Jan 26, 2017 at 7:56 PM, Sean Silva <chiso...@gmail.com> wrote:

On Tue, Jan 24, 2017 at 11:29 AM, Rui Ueyama <ru...@google.com> wrote:

Well, maybe, we should just change the Linux kernel instead of tweaking our tokenizer too hard.

This is silly. Writing a simple and maintainable lexer is not hard (look e.g. at https://reviews.llvm.org/D10817). There are some complicated context-sensitive cases in linker scripts that break our approach of tokenizing up front (so we might want to hold off on), but we aren't going to die from implementing enough to lex basic arithmetic expressions independent of whitespace.

Hmm..., the crux of not being able to lex arithmetic expressions seems to be due to lack of context sensitivity. E.g. consider `foo*bar`. Could be a multiplication, or could be a glob pattern.

Looking at the code more closely, adding context sensitivity wouldn't be that hard. In fact, our ScriptParserBase class is actually a lexer (look at the interface; it is a lexer's interface). It shouldn't be hard to change from an up-front tokenization to a more normal lexer approach of scanning the text for each call that wants the next token. Roughly speaking, just take the body of the for loop inside ScriptParserBase::tokenize and add a helper which does that on the fly and is called by consume/next/etc. Instead of an index into a token vector, just keep a `const char *` pointer that we advance.

Once that is done, we can easily add a `nextArithmeticToken` or something like that which just lexes with different rules.

Implementing a linker is much harder than implementing a lexer. If we give our users the impression that implementing a compatible lexer is hard for us, what impression will we give them about the linker's implementation quality? If we can afford 100 lines of self-contained code to implement a concurrent hash table; we can afford 100 self-contained lines to implement a context-sensitive lexer. This is end-user visible functionality; we should be careful skimping on it in the name of simplicity.

-- Sean Silva

[Sean Silva via llvm-dev]

On Tue, Jan 24, 2017 at 7:57 AM, George Rimar <gri...@accesssoftek.com> wrote:

The lexical format of linker scripts requires a context-sensitive lexer.

Look at how gold does it. IIRC there are 3 cases that are something like: one is for file-name like things, one is for numbers and stuff, and the last category is for numbers and stuff but numbers can also include things like `10k` (I think; would need to look at the code to remember for sure). It's done in a very elegant way in gold (passing a callback "can continue" that says which characters can continue the token). Which token regex to use is dependent on the grammar production (hence context sensitive). If you look at the other message I sent in this thread just now, ScriptParserBase is essentially a lexer interface and can be pretty easily converted to a more standard on-the-fly character-scanning implementation of a lexer. Once that is done adding a new method to scan a different kind of token for certain parts of the parser.

-- Sean Silva

 

George.

[George Rimar via llvm-dev]

>The lexical format of linker scripts requires a context-sensitive lexer.

>

>Look at how gold does it. IIRC there are 3 cases that are something like: one is for file-name like things, one is for numbers and stuff, and the last category is for >numbers and stuff but numbers can also include things like `10k` (I think; would need to look at the code to remember for sure). It's done in a very elegant way in gold >(passing a callback "can continue" that says which characters can continue the token). Which token regex to use is dependent on the grammar production (hence >context sensitive). If you look at the other message I sent in this thread just now, ScriptParserBase is essentially a lexer interface and can be pretty easily converted to >a more standard on-the-fly character-scanning implementation of a lexer. Once that is done adding a new method to scan a different kind of token for certain parts of >the parser.

>

>-- Sean Silva

I think that approach should work and should not be hard to implement.

Though when I think about that feature from "end user POV" I wonder how much users of it can be ? AFAIK we have the only script found in the wild that suffers from absence of whitespaces in math expressions. Looks 99.9% of scripts are free of that issue. And writing "5*6" instead "5 * 6" is looks not nice from code style.

Adding more code to LLD requires additional support for it at the end.

I am not going to say we should or should not doing that, that is just my concern. Moreover I probably would try to do that (just in case, to extend flexibility), though I can't say I see read need for that atm, basing on said above.

George.

[Sean Silva via llvm-dev]

Most of the features in the linker are for a single user at the time that they are implemented; but we know that we want that single user to work and so it doesn't matter. If the programs are buggy (not following ELF spec or whatever) then it may make sense to push for a fix upstream. But asking a user to change their program just because we can't be bothered to implement something simple (and clearly "correct") does not reflect well on the LLD project.

-- Sean Silva

 

George.

[Rafael Avila de Espindola via llvm-dev]

> Hmm..., the crux of not being able to lex arithmetic expressions seems to
> be due to lack of context sensitivity. E.g. consider `foo*bar`. Could be a
> multiplication, or could be a glob pattern.
>
> Looking at the code more closely, adding context sensitivity wouldn't be
> that hard. In fact, our ScriptParserBase class is actually a lexer (look at
> the interface; it is a lexer's interface). It shouldn't be hard to change
> from an up-front tokenization to a more normal lexer approach of scanning
> the text for each call that wants the next token. Roughly speaking, just
> take the body of the for loop inside ScriptParserBase::tokenize and add a
> helper which does that on the fly and is called by consume/next/etc.
> Instead of an index into a token vector, just keep a `const char *` pointer
> that we advance.
>
> Once that is done, we can easily add a `nextArithmeticToken` or something
> like that which just lexes with different rules.

I like that idea. I first thought of always having '*' as a token, but
then space has to be a token, which is an incredible pain.

I then thought of having a "setLexMode" method, but the lex mode can
always be implicit from where we are in the parser. The parser should
always know if it should call next or nextArithmetic.

And I agree we should probably implement this. Even if it is not common,
it looks pretty silly to not be able to handle 2*5.

Cheers,
Rafael

[Rui Ueyama via llvm-dev]

Sean,

So as you noticed that linker script tokenization rule is not very trivial -- it is context sensitive. The current lexer is extremely simple and almost always works well. Improving "almost always" to "perfect" is not high priority because we have many more high priority things, but I'm fine if someone improves it. If you are interested, please take it. Or maybe I'll take a look at it. It shouldn't be hard. It's probably just a half day work.

As far as I know, the grammar is LL(1), so it needs only one push-back buffer. Handling INCLUDE directive can be a bit tricky though.

Maybe we should rename ScriptParserBase ScriptLexer.

[Sean Silva via llvm-dev]

On Fri, Jan 27, 2017 at 1:31 PM, Rui Ueyama <ru...@google.com> wrote:

Sean,

So as you noticed that linker script tokenization rule is not very trivial -- it is context sensitive. The current lexer is extremely simple and almost always works well. Improving "almost always" to "perfect" is not high priority because we have many more high priority things, but I'm fine if someone improves it. If you are interested, please take it. Or maybe I'll take a look at it. It shouldn't be hard. It's probably just a half day work.

Yeah. To be clear, I wasn't saying that this was high priority. Since I'm complaining so much about it maybe I should take a look this weekend :)

 

As far as I know, the grammar is LL(1), so it needs only one push-back buffer. Handling INCLUDE directive can be a bit tricky though.

Maybe we should rename ScriptParserBase ScriptLexer.

That sounds like a good idea.

-- Sean Silva

[Dmitry Golovin via llvm-dev]

At this point I'm able to link Linux kernel with LLD and objcopy doen't give me any errors.

 

The versions are:

 

Linux 4.10.0-rc5 (+ applied the patch from my previous message)

LLD 5.0.0 (https://github.com/llvm-mirror/lld db83a5cc3968b3aac1dbe3270190bd3282862e74) (+ applied D28612)

GNU objcopy (GNU Binutils) 2.27

 

The problem is that the resulting kernel doesn't boot. Does anybody have any suggestions on how to debug it or any guesses what did go wrong while linking?

 

Regards,

Dmitry

28.01.2017, 05:48, "Sean Silva via llvm-dev" <llvm...@lists.llvm.org>:

,

[George Rimar via llvm-dev]

>At this point I'm able to link Linux kernel with LLD and objcopy doen't give me any errors.

> 

>The versions are:

> 

>Linux 4.10.0-rc5 (+ applied the patch from my previous message)

>LLD 5.0.0 (https://github.com/llvm-mirror/lld db83a5cc3968b3aac1dbe3270190bd3282862e74) (+ applied D28612)

>GNU objcopy (GNU Binutils) 2.27

> 

>The problem is that the resulting kernel doesn't boot. Does anybody have any suggestions on how to debug it or any guesses what did go wrong while linking?

> 

>Regards,

>Dmitry

It should not boot atm, I believe.
I mentioned earlier, LLD currently generates wrong output for scripts like:

.rodata : {
 *(.rodata)
 *(.rodata.*)
 . = ALIGN(16);
 video_cards = .;
 *(.videocards)
 video_cards_end = .;​

That is sample from kernel realmode script. We produce wrong values for video_cards/video_cards_end.

Reduced sample is D29217, and thread with possible patch for that is D27415 which is under discussions now.

(Though there are also probably can be other issues, but that one is obvious atm).

I have a question also. You added -m elf_i386 to workaround emulation conflict issue in LLD, do you know

does output produced by BFD boot fine after that change ?

George.

[Dmitry Golovin via llvm-dev]

> I have a question also. You added -m elf_i386 to workaround emulation conflict issue in LLD, do you know

> does output produced by BFD boot fine after that change ?

Doesn't seem to affect BFD at all.

29.01.2017, 10:18, "George Rimar" <gri...@accesssoftek.com>:

[George Rimar via llvm-dev]

>> I have a question also. You added -m elf_i386 to workaround emulation conflict issue in LLD, do you know

>> does output produced by BFD boot fine after that change ?

>Doesn't seem to affect BFD at all.

Thanks !

​George.

[Alexander Benikowski via llvm-dev]

I am unknown to all the stuff happening in and around LLVM, as i am interested in the project but had not time to dig in, yet. Just reading here and there a bit. But i wanted to share my 2 cents on the Lexer topic. I have written multiple lexer/parser for private purpose and recently one for our product to compile Mathematical expressions into an internal format for evaluation.

So far i usually write a Lexer which does the tokenizing on very basic rules. What is a number, what is a single symbolor a whole Identifier, and what to skip(like spaces). Each token has metadata like relative offset to the previous one. A Parser on top uses the lexer to process the Tokens. My approach of storing the relative offsets allows to regroup tokens to single symbols respecting whitespaces at a later time.

Let's say we have(without quotes ofcourse)
"2 * 3"

The lexer always produces 3 tokens, 2 with the content and identification for a numeraical value(for 2 and 3) and one for an operator token. I can now either go for it and use these tokens to process as part of a mathematical evaluation or reconstruct the original stream. However this means that the stream below:

"(foo)"

is always lexed to 3 tokens (2 for the brackets and one for the identifier). The Parser then has the context on how to process these tokens.

My example has some drawbacks. By always splitting into the basic tokens and possibly regrouping you'll need more rellocations for processing. And therefore losing performance. However i find this approach simple to test and write tests for.

[Joerg Sonnenberger via llvm-dev]

On Tue, Jan 31, 2017 at 12:28:37PM +0100, Alexander Benikowski via llvm-dev wrote:
> Let's say we have(without quotes ofcourse)
> "2 * 3"
> The lexer always produces 3 tokens, 2 with the content and identification
> for a numeraical value(for 2 and 3) and one for an operator token.

The problem with the linker script grammar is that "foo *" and "foo*"
can't both tokenize to IDENTIFIER STAR, since in the context of a
wildcard expression they mean different things. It certainly can be
dealt with by making the tokens whitespace sensitive or doing post
processing, but it is certainly not a nice grammar.

Joerg

[Sean Silva via llvm-dev]

On Sat, Jan 28, 2017 at 9:57 AM, Dmitry Golovin <di...@golovin.in> wrote:

At this point I'm able to link Linux kernel with LLD and objcopy doen't give me any errors.

 

The versions are:

 

Linux 4.10.0-rc5 (+ applied the patch from my previous message)

LLD 5.0.0 (https://github.com/llvm-mirror/lld db83a5cc3968b3aac1dbe3270190bd3282862e74) (+ applied D28612)

GNU objcopy (GNU Binutils) 2.27

 

The problem is that the resulting kernel doesn't boot. Does anybody have any suggestions on how to debug it or any guesses what did go wrong while linking?

Based on our experience getting FreeBSD working, we spent most time getting the bootloader to accept the kernel.

To debug this, we mostly used two approaches:

- printf debugging in the bootloader (will require rebuilding the bootloader multiple times)

- using objdump-like tools to look at the differences between a good (BFD or gold linked) kernel and the failing (LLD-linked) kernel. (e.g. different program header, different section contents in certain sections that the bootloader looks at, etc.)

As far as the setup, I would recommend setting up qemu for actually running the LLD-linked kernel and custom bootloader etc. because then you can have a single script that rebuilds the bootloader and kernel and copies the files to the VM. This reduces iteration time significantly.

Davide is the one that set that up and could probably provide more details, but qemu docs might be good enough that you can set things up without much effort (not sure though).

-- Sean Silva

[Sean Silva via llvm-dev]

On Sat, Jan 28, 2017 at 9:57 AM, Dmitry Golovin <di...@golovin.in> wrote:

At this point I'm able to link Linux kernel with LLD and objcopy doen't give me any errors.

 

The versions are:

 

Linux 4.10.0-rc5 (+ applied the patch from my previous message)

LLD 5.0.0 (https://github.com/llvm-mirror/lld db83a5cc3968b3aac1dbe3270190bd3282862e74) (+ applied D28612)

GNU objcopy (GNU Binutils) 2.27

 

The problem is that the resulting kernel doesn't boot. Does anybody have any suggestions on how to debug it or any guesses what did go wrong while linking?

As far as different things that can go wrong, some things to consider:

- LLD's output binary has the same (or similar) data as the BFD/gold output binary. E.g. if the LLD binary is only half as big (or the PT_LOAD's that the booloader looks at are half as bit), LLD might not be putting things into the output or into the right output sections.

- The bootloader is looking for something in the dynamic symbol table, but it isn't there. LLD might be resolving symbols differently.

- Section contents are different between LLD and BFD/gold. E.g. for freebsd there is a "linker set" section which contains pointers to a bunch of metadata structs that are needed. LLD was not relocating these correctly because the symbols were not ending up in the output or something like that (I forget exactly; Michael might remember better).

-- Sean Silva

[George Rimar via llvm-dev]

>As far as the setup, I would recommend setting up qemu for actually running the LLD-linked kernel and custom bootloader etc. because then you can have a single >script that rebuilds the bootloader and kernel and copies the files to the VM. This reduces iteration time significantly.

>Davide is the one that set that up and could probably provide more details, but qemu docs might be good enough that you can set things up without much effort 

>(not sure though).

>

>-- Sean Silva

 ​

By the way, yesterday I configured "smallest possible kernel", linked it with BFD and launched under QEMU.

It is very small and takes a few seconds to build it from scratch for me, used next article:

http://mgalgs.github.io/2015/05/16/how-to-build-a-custom-linux-kernel-for-qemu-2015-edition.html

Now I am going to link it with LLD and check if it boots or now. 

I think that should be fastest way - boot that little core and then enable features

one by one or group by group and fix other things on the road.

Previously I also worked on a patches for kernel but did not try to minimize it and used some default configuration,

what probably was good for finding mutliple issues from all sides, but not ideal way to fix/test startup and things.

George.

[George Rimar via llvm-dev]

> I still didn't do anything with "Setup too big!" problem, just commented out the assert.
>

So I investigated this today.
(It is about https://github.com/torvalds/linux/blob/master/arch/x86/boot/setup.ld#L52)

Script ends with:
.bss :
{
__bss_start = .;
*(.bss)
__bss_end = .;
}
. = ALIGN(16);
_end = .;

/DISCARD/ : { *(.note*) }

. = ASSERT(_end <= 0x8000, "Setup too big!");

And what LLD do here. It makes command list to be:
Place .bss, Place non-allocated orphans, Assign _end.
So places command "_end = " after commands for orphan sections.
There are plenty of them: debug_*, .comment, .shstrtab, .symtab, .strtab.
And _end value is set to not what expected finally.

Fix on script side can be very easy, just add one more line before "/DISCARD/":
.debug_info : { *(.debug_info) }
That way LLD will attach that non-allocated orphans after .debug_info section.
And _end will be calculated earlier and be correct.

I just had a quick look about how to fix that. We have shouldSkip(const BaseCommand &Cmd) with comment:
// We don't want to go over alignments, since doing so in
// rx_sec : { *(rx_sec) }
// . = ALIGN(0x1000);
// /* The RW PT_LOAD starts here*/
// rw_sec : { *(rw_sec) }
// would mean that the RW PT_LOAD would become unaligned.

I think probably fix could be to allow go over alignments if we placing non-allocatable orphans,
them are placed the last and should not cause such issue to happen.

[George Rimar via llvm-dev]

>I think probably fix could be to allow go over alignments if we placing non-allocatable orphans,
>them are placed the last and should not cause such issue to happen.
>
>George.

That will not work so simple. I'll take another look.

[Sean Silva via llvm-dev]

On Thu, Feb 2, 2017 at 12:38 AM, George Rimar <gri...@accesssoftek.com> wrote:

>As far as the setup, I would recommend setting up qemu for actually running the LLD-linked kernel and custom bootloader etc. because then you can have a single >script that rebuilds the bootloader and kernel and copies the files to the VM. This reduces iteration time significantly.

>Davide is the one that set that up and could probably provide more details, but qemu docs might be good enough that you can set things up without much effort 

>(not sure though).

>

>-- Sean Silva

 ​

By the way, yesterday I configured "smallest possible kernel", linked it with BFD and launched under QEMU.

It is very small and takes a few seconds to build it from scratch for me, used next article:

http://mgalgs.github.io/2015/05/16/how-to-build-a-custom-linux-kernel-for-qemu-2015-edition.html

Now I am going to link it with LLD and check if it boots or now. 

I think that should be fastest way - boot that little core and then enable features

one by one or group by group and fix other things on the road.

My experience with linker bugs is that usually when things are mis-linked, they are in the "core". E.g. startup code. So linking a small kernel may not avoid as many bugs as you expect. For example, for FreeBSD, I don't think we hit any issues in anything that could have been configured out.

-- Sean Silva

[Dmitry Golovin via llvm-dev]

I have just checked it, the startup.elf and realmode.elf are fine. Only few changes are required for mainline kernel and one commit has to be reverted from lld and a few patches have to be applied.

 

The only step when I have used BFD is linking vmlinux. I have manually set LD variable in vmlinux_link() function. The vmlinux produced by lld doesn't work yet. I will compare it to the one produced by GNU ld and try to figure out what is wrong (maybe you can suggest some useful objdump flags?)

 

Regards,

Dmitry

03.02.2017, 02:23, "Sean Silva" <chiso...@gmail.com>:

[Sean Silva via llvm-dev]

On Thu, Feb 2, 2017 at 4:35 PM, Dmitry Golovin <di...@golovin.in> wrote:

I have just checked it, the startup.elf and realmode.elf are fine. Only few changes are required for mainline kernel and one commit has to be reverted from lld and a few patches have to be applied.

 

The only step when I have used BFD is linking vmlinux. I have manually set LD variable in vmlinux_link() function. The vmlinux produced by lld doesn't work yet. I will compare it to the one produced by GNU ld and try to figure out what is wrong (maybe you can suggest some useful objdump flags?)

With objdump I would recommend looking at program headers. In particular at PT_LOAD's and the dynamic symbol table. Anything in the dynamic table is also worth scrutinizing. One thing to keep an eye out for is addresses/offsets that look "weird"; e.g. maybe the LLD version thinks a symbol has address 0 or some insane value, vs BFD/gold which has a more sane value.

Also, set up your system so that you rebuild/reinstall the bootloader too so that you can add printf's in there to hone in on where the boot is going wrong. The following workflow might be useful:

Step 1: add a printf to the bootloader to try to hone in on the exact place where things are going wrong

Step 2: rebuild/reinstall/reboot the new bootloader with the LLD-linked kernel

Step 3: boot and observe the print's (or maybe things crashed before reaching your print, which is just as useful to know)

Step 4: think about what you observed in Step 3, then go to Step 1, using these results to inform the next set of print's to add

With appropriate scripts (and a nice qemu setup), one iteration of this may take 10 minutes (say). You may have to repeat it (say) 20 times to pinpoint the exact place where things are going wrong (e.g. "the bootloader is crashing in the memcpy for the second PT_LOAD" or "the boot is failing because the bootloader is reading from a bogus address that it got from this part of the binary"). That is 200 minutes which isn't too bad.

One thing to keep in mind is that this is not like debugging a race condition or other nasty nondeterministic bug. This should be quite deterministic so you just have to be systematic and keep narrowing down until you find where things go wrong. It just requires determination.

Once narrowed-down, you should hopefully have a clear indication of where to look in the binary and compare with gold/bfd and hopefully the discrepancy is pretty clear. Then you "just" need to figure out why LLD produces this result and what to change to avoid the problem.

One amazing tool if you are working with object files is "010 Editor" https://www.sweetscape.com/010editor/ with a "binary template" for ELF files. I think there is an ELF "binary template" for 010 Editor floating around the net, but the best one is Michael's one that he has evolved over the years (ask him for it). If you haven't done so already, I recommend that you sit down at Michael's desk one day and work with him to debug one of these nasty "what is wrong with this binary and why?" issues so you can see him do his thing; he's amazingly good at it.

Also, if you need a quick refresher about this x86 boot stuff (to be somewhat oriented about the environment in which all this stuff is happening), you may want to skim:

http://duartes.org/gustavo/blog/post/how-computers-boot-up/

http://duartes.org/gustavo/blog/post/kernel-boot-process/

http://duartes.org/gustavo/blog/post/memory-translation-and-segmentation/

http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and-protection/

-- Sean Silva

[George Rimar via llvm-dev]

​​>On Fri, Jan 27, 2017 at 1:31 PM, Rui Ueyama <ru...@google.com> wrote:

​>>Sean,
​>>

>>So as you noticed that linker script tokenization rule is not very trivial -- it is context sensitive. The current lexer is extremely >>simple and almost always works well. Improving "almost always" to "perfect" is not high priority because we have many more >>high priority things, but I'm fine if someone improves it. If you are interested, please take it. Or maybe I'll take a look at it. It >>shouldn't be hard. It's probably just a half day work.

>

>Yeah. To be clear, I wasn't saying that this was high priority. Since I'm complaining so much about it maybe I should take a look >this weekend :)​

>

​>>As far as I know, the grammar is LL(1), so it needs only one push-back buffer. Handling INCLUDE directive can be a bit tricky >>though.
​>>

>>Maybe we should rename ScriptParserBase ScriptLexer.

>

>That sounds like a good idea.

>

>-- Sean Silva

Just in case, patch implementing this ideas is D29576. Works fine.

Imho looks fine either, except part that switches lexer modes. Probably I can impove it somehow if overall

direction is ok.

George.

 

[George Rimar via llvm-dev]

>I have just checked it, the startup.elf and realmode.elf are fine. Only few changes are required for mainline kernel and one >commit has to be reverted from lld and a few patches have to be applied.

> 

>The only step when I have used BFD is linking vmlinux. I have manually set LD variable in vmlinux_link() function. The vmlinux >produced by lld doesn't work yet. I will compare it to the one produced by GNU ld and try to figure out what is wrong (maybe >you can suggest some useful objdump flags?)

> 

>Regards,

>Dmitry

Just want to share latest results of investigation from my side. 

I traced kernel linked with LLD to find where it fails.

LLD linked kernel starts execution and then I came up to protected_mode_jump​ function, which intention to jump to startup_64:

    jmpl *%eax # Jump to the 32-bit entrypoint

(https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/pmjump.S#L76)

(https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/kernel/head_64.S#L48)

It does not happen. Code executes right before jmpl and then fail on this call for me, so startup_64 never called.

startup_64 is a part of vmlinux binary. So as you said vmlinux has troubles and after doing readelf -a on LLD linked and bfd linked ones, I found that LLD outputs vmlinux as executable and bfd output is DSO. Had no chance to investigate why that happens, but pretty sure that is the not fine.

Invocation line for us does not contain -shared:

-m elf_x86_64

--script home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/vmlinux.lds

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/head_64.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/misc.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/string.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/cmdline.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/error.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/piggy.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/cpuflags.o

-o vmlinux

George.

[Sean Silva via llvm-dev]

On Wed, Feb 8, 2017 at 7:59 AM, George Rimar <gri...@accesssoftek.com> wrote:

>I have just checked it, the startup.elf and realmode.elf are fine. Only few changes are required for mainline kernel and one >commit has to be reverted from lld and a few patches have to be applied.

> 

>The only step when I have used BFD is linking vmlinux. I have manually set LD variable in vmlinux_link() function. The vmlinux >produced by lld doesn't work yet. I will compare it to the one produced by GNU ld and try to figure out what is wrong (maybe >you can suggest some useful objdump flags?)

> 

>Regards,

>Dmitry

Just want to share latest results of investigation from my side. 

I traced kernel linked with LLD to find where it fails.

LLD linked kernel starts execution and then I came up to protected_mode_jump​ function, which intention to jump to startup_64:

    jmpl *%eax # Jump to the 32-bit entrypoint

(https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/pmjump.S#L76)

(https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/kernel/head_64.S#L48)

It does not happen. Code executes right before jmpl and then fail on this call for me, so startup_64 never called.

That address seems to come from here: https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/pm.c#L124

```

protected_mode_jump(boot_params.hdr.code32_start,

   (u32)&boot_params + (ds() << 4));

```

That boot_params.hdr.code32_start field is probably either invalid (bad reloc or something else causing the bootloader to calculate the wrong address) or valid but the thing it thinks it is pointing to wasn't loaded (missing PT_LOAD etc.). 

Btw, how did you narrow it down to that specific instruction? That's pretty handy.

 

startup_64 is a part of vmlinux binary. So as you said vmlinux has troubles and after doing readelf -a on LLD linked and bfd linked ones, I found that LLD outputs vmlinux as executable and bfd output is DSO. Had no chance to investigate why that happens, but pretty sure that is the not fine. 

Invocation line for us does not contain -shared:

-m elf_x86_64

--script home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/vmlinux.lds

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/head_64.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/misc.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/string.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/cmdline.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/error.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/piggy.o

home/umb/linux_kernel/linux/linux/arch/x86/boot/compressed/cpuflags.o

-o vmlinux

I think you can also get DSO with -pie I think, but I don't see that either. This is quite mysterious. I also did a quick look at the linker script and didn't see anything at first glance that would cause DSO output (can linker scripts even control EType?). The bootloader might not even look at the EType though.

-- Sean Silva

George.

[George Rimar via llvm-dev]

>That address seems to come from >here: https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/pm.c#L124

>

>```

>protected_mode_jump(boot_params.hdr.code32_start,

>   (u32)&boot_params + (ds() << 4));

>```

>

>That boot_params.hdr.code32_start field is probably either invalid (bad reloc or something else causing the bootloader to >calculate the wrong address) or valid but the thing it thinks it is pointing to wasn't loaded (missing PT_LOAD etc.). 

boot_params.hdr.code32_start field is valid :) It is 0x100000, like expected (btw thanks for those links on info how kernel boots, they were pretty useful). I checked it is valid using trace:

void go_to_protected_mode(void)

{

if (boot_params.hdr.code32_start == 0x100000)

  puts("go_to_protected_mode 1\n");

else

  puts("go_to_protected_mode 2\n");

while (1) {};

I had to use infinite loop here, because QEMU does not crash for me, but do domething what looks like reboot and clears all my traces output. Infinite loop helps to see traces text, if placed before dead point.

>

>Btw, how did you narrow it down to that specific instruction? That's pretty handy.

I used very simple approach, did not know how to do that properly :), so inserted infinite loops in asm code:

foo:

 jmp foo

And watched for behavior of QEMU. If it just hanged that was fine, I knew I am inside my loop, if QEMU rebooted, I knew it crashed at point I was looking for. So tracing bfd linked kernel and using documentation I found that startup_64 ​is next destination POI, and found that this instruction is the last before QEMU reboots for me.

 

>I think you can also get DSO with -pie I think, but I don't see that either. This is quite mysterious. I also did a quick look at the >linker script and didn't see anything at first glance that would cause DSO output (can linker scripts even control EType?). The >bootloader might not even look at the EType though.

>

>-- Sean Silva

I think best way would be to look at what is invocation for BFD here. I am pretty sure -shared/-pie flag is just lost because of some configuration issue, probably it checks that we are running bfd may be. Just a guess.

George.

[George Rimar via llvm-dev]

>>I think you can also get DSO with -pie I think, but I don't see that either. This is quite mysterious. I also did a quick look at the >>linker script and didn't see anything at first glance that would cause DSO output (can linker scripts even control EType?). The >>bootloader might not even look at the EType though.

>>

>>-- Sean Silva

>
>I think best way would be to look at what is invocation for BFD here. I am pretty sure -shared/-pie flag is just lost because of >some configuration issue, probably it checks that we are running bfd may be. Just a guess.

>

>George.

Because reproduce linked with bfd also produced executable for me.

George.

[Sean Silva via llvm-dev]

On Thu, Feb 9, 2017 at 1:27 AM, George Rimar <gri...@accesssoftek.com> wrote:

>That address seems to come from >here: https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/pm.c#L124

>

>```

>protected_mode_jump(boot_params.hdr.code32_start,

>   (u32)&boot_params + (ds() << 4));

>```

>

>That boot_params.hdr.code32_start field is probably either invalid (bad reloc or something else causing the bootloader to >calculate the wrong address) or valid but the thing it thinks it is pointing to wasn't loaded (missing PT_LOAD etc.). 

boot_params.hdr.code32_start field is valid :) It is 0x100000, like expected

Then I suspect that that segment isn't being loaded. Is there a PT_LOAD that covers that address? Is the bootloader loading it?

 

(btw thanks for those links on info how kernel boots, they were pretty useful).

I'm glad you found it useful! Those articles are amazing.

 

I checked it is valid using trace:

void go_to_protected_mode(void)

{

if (boot_params.hdr.code32_start == 0x100000)

  puts("go_to_protected_mode 1\n");

else

  puts("go_to_protected_mode 2\n");

while (1) {};

I had to use infinite loop here, because QEMU does not crash for me, but do domething what looks like reboot and clears all my traces output. Infinite loop helps to see traces text, if placed before dead point.

>

>Btw, how did you narrow it down to that specific instruction? That's pretty handy.

I used very simple approach, did not know how to do that properly :), so inserted infinite loops in asm code:

foo:

 jmp foo

I suspected something like this :) I've had to do that in the past. btw if you are ever in a tight spot, a good thing to remember is EB FE (jmp -2) which encodes an infinite loop on x86 (all modes, even 16bit I believe). It is small so can easily be patched over most instruction using a hex editor if necessary.

-- Sean Silva

[George Rimar via llvm-dev]

>Then I suspect that that segment isn't being loaded. Is there a PT_LOAD that covers that address? Is the bootloader loading it?

I had had to switch to an another task for now and did not had chance to investigate deeper this issue yet.

I'll let you know if have something new here.

>I suspected something like this :) I've had to do that in the past. btw if you are ever in a tight spot, a good thing to remember is EB FE (jmp -2) which encodes an >infinite loop on x86 (all modes, even 16bit I believe). It is small so can easily be patched over most instruction using a hex editor if necessary.

>

>-- Sean Silva

Nice. I saw this 2 byte jump it was used in 16 bits mode when did something for R_386_PC8 relocations:

https://github.com/torvalds/linux/blob/master/arch/x86/boot/header.S#L297

George.

[George Rimar via llvm-dev]

>I think you can also get DSO with -pie I think, but I don't see that either. This is quite mysterious. I also did a quick look at the >linker script and didn't see anything at first glance that would cause DSO output (can linker scripts even control EType?). The >bootloader might not even look at the EType though.

>

>-- Sean Silva

Place is:

https://github.com/torvalds/linux/blob/master/arch/x86/boot/compressed/Makefile#L51

It checks if -z noreloc-overflow is available and adds -pie and other flags in that case.

If I comment out this lines:

#LDFLAGS += $(shell $(LD) --help 2>&1 | grep -q "\-z noreloc-overflow" \

# && echo "-z noreloc-overflow -pie --no-dynamic-linker")​

Then as expected bfd linked vmlinux also becomes executable. And it is even still boots for me.

George.

[George Rimar via llvm-dev]

>>That boot_params.hdr.code32_start field is probably either invalid (bad reloc or something else causing the bootloader to >>calculate the wrong address) or valid but the thing it thinks it is pointing to wasn't loaded (missing PT_LOAD etc.). 

>boot_params.hdr.code32_start field is valid :) It is 0x100000, like expected

>

>Then I suspect that that segment isn't being loaded. Is there a PT_LOAD that covers that address? Is the bootloader loading it?

 

That issue is gone. Not sure what changed, but looks something was fixed in LLD during last week.

Latest status of booting linux kernel is next currently:

At this location,

https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/head_64.S#L424

kernel calls extract_kernel(). And 2 lines below it tries to jmp to the address of decompressed kernel and fails to do that for me.

extract_kernel() method is:

https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L334

I added next lines before return:

__putstr("hi from extract_kernel");

if ((int) output == 0x1000000)

  __putstr("== 0x1000000");

output[0] = 0xEB;

output[1] = 0xFE;

return output;

And during boot in shows all text from above and enters infinite loop as expected. So, that means it successfully jumps to 0x1000000, but looks something is wrong in decompressed code. Next destination point should be startup_64​.

https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/kernel/head_64.S#L50

Though as I mentioned it does not reach it for me.

Next step I probably going to do is dump/printf-trace that memory area of decompressed kernel to compare with what produced there when BFD is used. Have no better ideas now.

George.

[George Rimar via llvm-dev]

>>>That boot_params.hdr.code32_start field is probably either invalid (bad reloc or something else causing the bootloader to >>>calculate the wrong address) or valid but the thing it thinks it is pointing to wasn't loaded (missing PT_LOAD etc.). 

>>boot_params.hdr.code32_start field is valid :) It is 0x100000, like expected

>>

>>Then I suspect that that segment isn't being loaded. Is there a PT_LOAD that covers that address? Is the bootloader loading it?

> 

>That issue is gone. Not sure what changed, but looks something was fixed in LLD during last week.

Looks what really happend is that previously I expected it should have jump to a different startup_64, since there are 2 of them :( 

https://github.com/torvalds/linux/blob/212f30008a284a9312d95dad6cc237ff81173d73/arch/x86/boot/compressed/head_64.S#L235

https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/kernel/head_64.S#L50

So looks real place of issue was that one I located today.

George.

[Sean Silva via llvm-dev]

Based on your other post and some googling, it seems like there are two separate kernel binaries. One is a "loader" binary that does the decompression and some simple dynamic loader/linker type behavior and the "real" kernel binary.

It seems based on your description that the "loader" binary is running fine. The bug is not likely to be corrupted data in the decompressed output (that is just calling into a gzip routine or something). You shouldn't have to dump/printf-trace from memory during boot to see that data since the "real" kernel binary that is being decompressed into that memory region is probably already somewhere in your build tree (arch/x86/boot/compressed/Makefile seems like it has the details; I think it is the file called `vmlinux`; also see this article with more info about all the pieces: https://en.wikipedia.org/wiki/Vmlinux). It should be possible to do some static analysis (ELF header, phdrs, and symbols/relocations) on the "real" kernel binary directly.

One place to start doing dynamic analysis is these two lines: https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L407

parse_elf(output);

handle_relocations(output, output_len, virt_addr);

parse_elf:

https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L268

This seems to implement a bare-bones loader:

``

for (i = 0; i < ehdr.e_phnum; i++) {

phdr = &phdrs[i];

switch (phdr->p_type) {

case PT_LOAD:

#ifdef CONFIG_RELOCATABLE

dest = output;

dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);

#else

dest = (void *)(phdr->p_paddr);

#endif

memmove(dest, output + phdr->p_offset, phdr->p_filesz);

break;

default: /* Ignore other PT_* */ break;

}

}`

```

Based on the fact that you are hitting the EB FE at the load address, this seems like it is working fine (though if there are multiple PT_LOAD's there may still be issues lurking with the other ones).

handle_relocations:

https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L173

Okay, this one looks likely to be where the issues are manifesting.

First off, it seems like they have some sort of homegrown relocation scheme:

https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L211

```

/*

* Process relocations: 32 bit relocations first then 64 bit after.

* Three sets of binary relocations are added to the end of the kernel

* before compression. Each relocation table entry is the kernel

* address of the location which needs to be updated stored as a

* 32-bit value which is sign extended to 64 bits.

*

* Format is:

*

* kernel bits...

* 0 - zero terminator for 64 bit relocations

* 64 bit relocation repeated

* 0 - zero terminator for inverse 32 bit relocations

* 32 bit inverse relocation repeated

* 0 - zero terminator for 32 bit relocations

* 32 bit relocation repeated

*

* So we work backwards from the end of the decompressed image.

*/

```

Grepping around, it seems like they build this list of relocations based on some sort of homegrown tooling in arch/x86/tools/. E.g. look at arch/x86/tools/relocs.c. The main exported function from there (modulo being compiled twice in 32/64 mode) is:

```

#if ELF_BITS == 64

# define process process_64

#else

# define process process_32

#endif 

void process(FILE *fp, int use_real_mode, int as_text,

    int show_absolute_syms, int show_absolute_relocs,

    int show_reloc_info)

{

regex_init(use_real_mode);

read_ehdr(fp);

read_shdrs(fp);

read_strtabs(fp);

read_symtabs(fp);

read_relocs(fp);

if (ELF_BITS == 64)

percpu_init();

if (show_absolute_syms) {

print_absolute_symbols();

return;

}

if (show_absolute_relocs) {

print_absolute_relocs();

return;

}

if (show_reloc_info) {

print_reloc_info();

return;

}

emit_relocs(as_text, use_real_mode);

}

```

There may be something that this program doesn't like about the LLD-linked "real" kernel executable. If regular static analysis of the "real" kernel binaries doesn't reveal anything wrong (I'm guessing it will though and is probably the best first thing to look at), you can try tracing the execution of this program on the LLD-linked executable and the BFD/gold executable and diff the results. arch/x86/boot/compressed/Makefile seems to have some details about the invocation of this `relocs` program.

Also, down the road once we have your current configuration working, it is probably worth toggling each of these kconfig options as they may tickle different LLD issues:

CONFIG_X86_NEED_RELOCS

CONFIG_RELOCATABLE

(this is just from a quick glance at the ifdef's in the compressed boot loader code)

-- Sean Silva

 

George.

[George Rimar via llvm-dev]

Thanks for your ideas, Sean !

>The bug is not likely to be corrupted data in the decompressed output (that is just calling into a gzip routine or something). You shouldn't have to dump/printf->trace from memory during boot to see that data since the "real" kernel binary that is being decompressed into that memory region is probably already somewhere > in your build tree (arch/x86/boot/compressed/Makefile seems like it has the details; 

Yeah, what I mean, it not just decompresses it, it also applies relocations to the decompressed output:

__decompress(input_data, input_len, NULL, NULL, output, output_len, NULL, error);

parse_elf(output);

handle_relocations(output, output_len, virt_addr);

My idea of quick dumping was based on a some experience for one of FreeBSD loaders I fixed earlier. It was very short (512 bytes if I am not mistaken) and just quick look on LLD and bfd linked hex views revealed the reason instantly. LLD linked binary was like corrupted with 4 bytes holes. All other code around was equal. That showed that we just did not apply one of relocations (R_386_16 I think) at all.

So main idea of dumping was to check if there is something obviously wrong. May be it was not the best idea :) I want to try though.

> Grepping around, it seems like they build this list of relocations based on some sort of homegrown tooling in arch/x86/tools/. E.g. look at arch/x86/tools/relocs.c. 

So I am also suspecting it is something relative to relocations, my suspicion also based on fact that we had to 

implement --emit-relocs option to support how they handle it.

This is new feature for LLD and can probably still may have bugs (last part was committed on this friday)​ and needs testing.

Details about how they use --emit-relocs was in next comment: https://reviews.llvm.org/D28612#647277. 

In short --emit-relocs generates .rel[a].xxx sections in output,

they extract these sections using objdump and wrap it into separate binary with own format. 

And use this data for self-doing relocatations.

George.

[George Rimar via llvm-dev]

I found that bug finally :)

I dumped output for kernel and observed that LLD output starts from ELF header, while BFD points to some code data it seems, what is expected since execution flow makes jmp there.

That means decompressed kernel was not copied properly and issue is at one of points you sent earlier:

https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L300

Looking on readelf output for vmlinux,  BFD loads has correct LMAs:

  LOAD           0x0000000000200000 0xffffffff81000000 0x0000000001000000

                 0x0000000000241000 0x0000000000241000  R E    200000

  LOAD           0x0000000000600000 0xffffffff81400000 0x0000000001400000

                 0x0000000000041000 0x0000000000041000  RW     200000

  LOAD           0x0000000000641000 0xffffffff81441000 0x0000000001441000

                 0x0000000000070699 0x00000000000c4000  RWE    200000

But LLD output is broken at that part, all LMA are 0x0000000080000000 here​:

  LOAD           0x0000000000001000 0xffffffff81000000 0x0000000080000000

                 0x0000000000242000 0x0000000000242000  R E    1000

  LOAD           0x0000000000243000 0xffffffff81400000 0x0000000080000000

                 0x0000000000041000 0x0000000000041000  RW     1000

  LOAD           0x0000000000284000 0xffffffff81441000 0x0000000080000000

                 0x0000000000071000 0x00000000000c4000  RWE    1000

That happened because script sets address in next way:

.text : AT(ADDR(.text) - 0xffffffff80000000) { ... }

.data : AT(ADDR(.data) - 0xffffffff80000000) { .. }

.init.begin : AT(ADDR(.init.begin) - 0xffffffff80000000) { ... }

We currently just calculate ADDR(..) as zero in such cases and because of that 

always have result == 0x0000000080000000 finally. That is the reason of kernel corruption.

I prepared patch that fixes it: https://reviews.llvm.org/D30163, with that in, I was able to pass that place

and QEMU no more reboots for me ! 

Though it still hangs a bit later, atm it shows:

CPU: AMD QEMU Virtual CPU version 2.5+ (family: 0x6, model: 0x6, stepping: 0x3)

Performance Events: PMU not available due to virtualization, using software events only.

..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1

..MP-BIOS bug: 8254 timer not connected to IO-APIC

...trying to set up timer (IRQ0) through the 8259A ...

..... (found apic 0 pin 2) ...

....... failed.

...trying to set up timer as Virtual Wire IRQ...

..... failed.

...trying to set up timer as ExtINT IRQ...

..... failed :(.

Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.

---[ end Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.

random: fast init done

but that is definetely a different new issue, I am investigating it.

(at least I am happy there is no more silent QEMU reboots and it shows some readable error messages it seems).

George.

[George Rimar via llvm-dev]

And I think current issue with "Kernel panic - not syncing: IO-APIC + timer doesn't work!" is also clear.

timer_irq_works(void) never returns 1:

https://github.com/torvalds/linux/blob/d966564fcdc19e13eb6ba1fbe6b8101070339c3d/arch/x86/kernel/apic/io_apic.c#L1641

I think it happens because of jiffies (http://www.makelinux.net/books/lkd2/ch10lev1sec3#ch10fig01)

It should have the same address as jiffies_64:

https://github.com/torvalds/linux/blob/b66484cd74706fa8681d051840fe4b18a3da40ff/arch/x86/kernel/vmlinux.lds.S#L41

And that is true for BFD linked binary:

 10595: ffffffff8140b000     8 OBJECT  GLOBAL DEFAULT    8 jiffies

 11730: ffffffff8140b000     8 OBJECT  GLOBAL DEFAULT    8 jiffies_64

 

But something is wrong with them for LLD case:

 6422: ffffffff8140b000     8 OBJECT  GLOBAL DEFAULT   19 jiffies

 7416: ffffffff81400000     0 NOTYPE  GLOBAL DEFAULT   19 jiffies_64

  

I think we probably incorrectly assign symbols outside SECTIONS declarations in scripts. Looking at it.

George.

[Dmitry Golovin via llvm-dev]

Hi,

 

I can't build the Linux kernel anymore. With the latest kernel source and latest LLVM and LLD (with D30163) I can only get this error message when linking vmlinux:

 

ld.lld: error: unable to move location counter backward

 

Did you have similar errors? How do I get rid of it?

 

Regards,

Dmitry

20.02.2017, 17:51, "George Rimar" <gri...@accesssoftek.com>:

[Rui Ueyama via llvm-dev]

Maybe that is https://reviews.llvm.org/D30171.

[Dmitry Golovin via llvm-dev]

Thank you! It worked and I finally got a working kernel. Here is the kernel boot log:

 

    Linux version 4.10.0+ (build@buildarch) (gcc version 6.3.1 20170109 (GCC) ) #2 Tue Feb 21 13:08:29 EET 2017

    Command line: console=ttyS0 noapic

    x86/fpu: Legacy x87 FPU detected.

    e820: BIOS-provided physical RAM map:

    BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable

    BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved

    BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved

    BIOS-e820: [mem 0x0000000000100000-0x0000000007fdffff] usable

    BIOS-e820: [mem 0x0000000007fe0000-0x0000000007ffffff] reserved

    BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved

    NX (Execute Disable) protection: active

    e820: last_pfn = 0x7fe0 max_arch_pfn = 0x400000000

    found SMP MP-table at [mem 0x000f6640-0x000f664f] mapped at [ffff8800000f6640]

    RAMDISK: [mem 0x04718000-0x07fdffff]

    Zone ranges:

      DMA32    [mem 0x0000000000001000-0x0000000007fdffff]

      Normal   empty

    Movable zone start for each node

    Early memory node ranges

      node   0: [mem 0x0000000000001000-0x000000000009efff]

      node   0: [mem 0x0000000000100000-0x0000000007fdffff]

    Initmem setup node 0 [mem 0x0000000000001000-0x0000000007fdffff]

    Intel MultiProcessor Specification v1.4

    MPTABLE: OEM ID: BOCHSCPU

    MPTABLE: Product ID: 0.1         

    MPTABLE: APIC at: 0xFEE00000

    Processor #0 (Bootup-CPU)

    IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23

    Processors: 1

    e820: [mem 0x08000000-0xfffbffff] available for PCI devices

    clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30582079743023 ns

    Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32169

    Kernel command line: console=ttyS0 noapic

    PID hash table entries: 512 (order: 0, 4096 bytes)

    Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)

    Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)

    Memory: 65016K/130552K available (740K kernel code, 218K rwdata, 116K rodata, 424K init, 236K bss, 65536K reserved, 0K cma-reserved)

    NR_IRQS:4352 nr_irqs:48 16

    console [ttyS0] enabled

    tsc: Fast TSC calibration using PIT

    tsc: Detected 2494.298 MHz processor

    Calibrating delay loop (skipped), value calculated using timer frequency.. 4988.59 BogoMIPS (lpj=9977192)

    pid_max: default: 4096 minimum: 301

    Mount-cache hash table entries: 512 (order: 0, 4096 bytes)

    Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)

    Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0

    Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0

    CPU: AMD QEMU Virtual CPU version 2.4.0 (family: 0x6, model: 0x6, stepping: 0x3)

    Performance Events: PMU not available due to virtualization, using software events only.

    APIC timer disabled due to verification failure

    clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns

    clocksource: Switched to clocksource refined-jiffies

    Unpacking initramfs...

    Freeing initrd memory: 58144K

    platform rtc_cmos: registered platform RTC device (no PNP device found)

    workingset: timestamp_bits=62 max_order=15 bucket_order=0

    Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled

    serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A

    Freeing unused kernel memory: 424K

    Write protecting the kernel read-only data: 4096k

    Freeing unused kernel memory: 1300K

    random: fast init done

    Freeing unused kernel memory: 1932K

    Failed to execute /init (error -2)

    Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.

    Kernel Offset: disabled

    ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.

 

Regards,

Dmitry

21.02.2017, 09:26, "Rui Ueyama" <ru...@google.com>:

[George Rimar via llvm-dev]

>Thank you! It worked and I finally got a working kernel. Here is the kernel boot log:

 

Sound great ! Though I ended with error from log below. Had no chance to fix it yet:

​

CPU: AMD QEMU Virtual CPU version 2.5+ (family: 0x6, model: 0x6, stepping: 0x3)

Performance Events: PMU not available due to virtualization, using software events only.

..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1

..MP-BIOS bug: 8254 timer not connected to IO-APIC

...trying to set up timer (IRQ0) through the 8259A ...

..... (found apic 0 pin 2) ...

....... failed.

...trying to set up timer as Virtual Wire IRQ...

..... failed.

...trying to set up timer as ExtINT IRQ...

..... failed :(.

Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.

---[ end Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.

random: fast init done

Did you see something alike ? Looks there is nothing in your log about these timers. I suppose you disabled those ?

George.

[Dmitry Golovin via llvm-dev]

Yes, I use the tinyconfig with only few options turned on. I also add 'noapic' to the kernel command line, so the booting process goes on. Unfortunately, the kernel is not yet capable of running init, but it is a great progress.

 

Regards,

Dmitry

21.02.2017, 13:25, "George Rimar" <gri...@accesssoftek.com>:

[Rui Ueyama via llvm-dev]

Wow, congratulations!