[LLVMdev] [LLD] support for dlltool generated libs in COFF/PECOFF

[Martell Malone]

Hi again rui, :)

I've got all the patches into llvm and clang for supporting mingw-w64 via compiler-rt and now we are able to build a full mingw-w64 toolchain without gcc :)

With great help from yaron and rnk.

I've CC'd them as they might have interest in seeing this target through with me to the end :) 

So I have again turned my attention to LLD so that we can also remove ld as a depend for this target 

Previously I raised the issue that while lld would infact link exe's for me the issue was the sections were malformed and thus crashed on launch.

I've done some debugging reguarding this and discovered it is due to differences between what MSVC's lib.exe and binutils dlltool produces

I have attached a very simple .def file and generated libs via dlltool along with an objdump -s log of the lib.exe version

I don't know too much about the internals of lld to be able to tell the exact cause of the problem but there are 3 specific differences I noted.

1. With dlltool each object has the name of a compiled .o file this is the one it generates when building the code section

With lib each object has the dll name 

2. With dlltool section 7 is used for the dll name and the function names with lib section 6 is used

3. Other differences are that there is no debug section with dlltool and the text section comes first rather than last with lib (that probably doesn't matter much though)

Please see attached notes.txt with objdump's of both libs

Can could you give me your thoughts on this and how best we support both layouts in COFF/PECOFF ?

What source files should I be looking at to patch this or is this something you can support with very little effort ?

I don't know if you would be opposed to supporting this or not being honest dlltool should generate libs using the same section number as lib.exe but we can't exactly change that now because existing toolchains and libs have already been built with this and changing that would break them.

If you are opposed I would like you to still tell me what I have to change to get it working after I do that I can narrow the scope for getting a patch up streamed into binutils to fix this :)

Kind Regards

Martell

[Martell Malone]

I forgot to attach the notes.txt with the objdump.

[Martell Malone]

After some more digging and creating a few testcases in lld I have narrowed it down to

The fact that dlltool generates

Contents of section .idata$7:
 0000 55534552 33322e64 6c6c0000           USER32.dll..

Where as lld expects

Contents of section .idata$6:
 0000 55534552 33322e64 6c6c0000           USER32.dll..

I recreated the hello64.test using dlltool for the lib and here is the asm dump of the final exe

hello64gnu.exe:      file format COFF-x86-64

Disassembly of section .text:
.text:
    3000:       ff 25 26 f0 ff ff       jmpq    *-4058(%rip)
    3006:       90      nop
    3007:       90      nop
    3008:       ff 25 26 f0 ff ff       jmpq    *-4058(%rip)
    300e:       90      nop
    300f:       90      nop
    3010:       48 83 ec 28     subq    $40, %rsp
    3014:       48 c7 c1 00 00 00 00    movq    $0, %rcx
    301b:       48 8d 15 e4 df ff ff    leaq    -8220(%rip), %rdx
    3022:       4c 8d 05 d7 df ff ff    leaq    -8233(%rip), %r8
    3029:       41 b9 00 00 00 00       movl    $0, %r9d
    302f:       e8 d4 ff ff ff  callq   -44
    3034:       b9 00 00 00 00  movl    $0, %ecx
    3039:       e8 c2 ff ff ff  callq   -62

So I am fairly certain this is the cause.

Many Thanks

Martell

[Martell Malone]

Hey guys,

So I was able to modify dlltool to produce the exact same layout as lib.exe with the same section numbers etc.

I've first managed to first create the correct section so that lld gives me link errors and then resolve those errors to create an exe.

This is one thing that is still missing that sticks out like a sore thumb

In the actual imports of the functions the objects are very different

As you can see lib.exe generates a very simple function import object.

  IMPORT_OBJECT[1-N]

    -> contains an IMPORT_OBJECT_HEADER and DATA

    data example -> _MessageBoxA.USER32.dll

dlltool however doesn't create an import object but infact has a complete IMAGE_FILE

  IMPORT_OBJECT[1-N]

    -> contains an IMAGE_FILE_HEADER and is a full object with SECTIONS

    the .text section has the stub code.

    the .idata$6 section has the name of the function

    it starts with 0x01 0x4D then MessageBoxA without an underscore

    The dll is not referenced and is assumed to be the one listed in the first object in .idata$6

This is where the actual problem lies because the dll functions never get resolved

Have you guys got any thoughts on weither we should support this or not within lld and if so how ?

There are some interesting things here

For example the gnu linker provides the section with the stub code and the offset for injection.

Should we be using the stub code provided in the text section or should we just ignore it and use the one in lld

They are the same anyway but its just interesting how the dlltool version gives us that option.

Kind Regards

Martell 

[Martell Malone]

For example the gnu linker provides the section with the stub code and the offset for injection.

Meant to say dlltool here not gnu linker :) 

[Ivan Garramona]

Hi Martell.

I too was trying to make the new COFF linker recognize the libraries generated by binutils, but it wasn't resolving the DLLs, just like you described. Apparently it is not identifying the libraries as import libraries but archives.

For testing purposes, i've managed to make the old COFF linker work with a simple test case. Had to make two little changes, and link the exe manually, because the linker doesn't understand the GNU flags, but it works. By the way, even the MS linker works with the binutils generated import libraries.

[Martell Malone]

Hi Ivan, :)

For testing purposes, I've managed to make the old COFF linker work with a simple test case

Have you got patches for this that I can look at ?

Had to make two little changes, and link the exe manually, because the linker doesn't understand the GNU flags

I've solved the issue of having a GNU driver for the linker so that's not an issue anymore

I sent in patches to the mailing list on this previously :)

I keep them updated over in the msys2 project btw

By the way, even the MS linker works with the binutils generated import libraries.

Yes i assume binutils was based from a very old version of a MS linker which did this layout at the time.

lld is so much newer and the MS linker has been updated to a more modern layout which is why we have this issue

Kind Regards

Martell

 

[Rui Ueyama]

Hi Martell,

Sorry for the belated response. I missed this email.

I took a quick look at the library file you have attached and noticed that the file consists of regular object files. LLD expects that all dllimported symbols are described using "short import library" (PE/COFF spec 7). Because I didn't test LLD with dlltool-style archive files, it's no surprise that it didn't work (although it should theoretically work).

Do you think that you can make dlltool to generate short import libraries? If you can, it would be the easiest way to solve the issue.

_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Martell Malone]

Hi Rui,

Thanks for getting back to me on this

Do you think that you can make dlltool to generate short import libraries? If you can, it would be the easiest way to solve the issue.

On my current tests ld seems to not like libraries built with short import libraries so this would require an ABI break in binutils.

dlltool has support for loads of legacy targets and changing this would also possibly break some of those targets so

Rather than working on dlltool I opted for creating a new tool for mingw-64 to generate libs from def files using the "short import library"

Another reason for me doing this is to have a complete clang based mingw-w64 toolchain without any gnu gcc or binutils dependencies.

It would be ideal however if we could support what dlltool currently produces in lld.

mingw.org toolchains will not be using the new tool I am creating and every distribution of mingw would have to rebuild their crt's just to work with lld.
This isn't ideal when clang is currenly used like a drop in add on for mingw-64 toolchains.

So if it is an easy enough fix we should support this in lld

Especially when MSVC already does.

Is there much involved with supporting this on the lld end?

Many Thanks

Martell

[Rui Ueyama]

It's doable to support dlltool-style import libraries by LLD. .idata sections in dlltool-style libraries are designed so that they will naturally construct the import descriptor table by just sorting them by section name. Supporting them is not hard.

One thing we need to do is that currently we construct the import descriptor directly from short import libraries. We don't create intermediate .idata$X sections from short import libraries. That means we cannot mix dlltool-style and short import libraries at this moment. We need to do something for that. Except that, supporting dlltool-style libraries should be straightforward.

[Martell Malone]

It's doable to support dlltool-style import libraries by LLD. .idata sections in dlltool-style libraries are designed so that they will naturally construct the import descriptor table by just sorting them by section name. Supporting them is not hard.

I don't really know my way around the code base of COFF well enough todo this myself.

If you can point me in the right direction as to what functions/sources to be looking at I think I can come up with a patch.

If you think you can do it fairly easily I can also create a test case for you if you want ?

One thing we need to do is that currently we construct the import descriptor directly from short import libraries. We don't create intermediate .idata$X sections from short import libraries. That means we cannot mix dlltool-style and short import libraries at this moment. We need to do something for that. Except that, supporting dlltool-style libraries should be straightforward.

 

This is a very unique scenario and I don't know of any tool that generates both styles of libraries.

I can create one if we need it for a test case but I think this would be overkill :) 

[Rui Ueyama]

On Sat, Jul 25, 2015 at 5:24 PM, Martell Malone <martel...@gmail.com> wrote:

It's doable to support dlltool-style import libraries by LLD. .idata sections in dlltool-style libraries are designed so that they will naturally construct the import descriptor table by just sorting them by section name. Supporting them is not hard.

I don't really know my way around the code base of COFF well enough todo this myself.

If you can point me in the right direction as to what functions/sources to be looking at I think I can come up with a patch.

If you think you can do it fairly easily I can also create a test case for you if you want ?

If it's not urgent, I can take a look after finishing up x86 COFF port and fixing up remaining x64 bugs.

One thing we need to do is that currently we construct the import descriptor directly from short import libraries. We don't create intermediate .idata$X sections from short import libraries. That means we cannot mix dlltool-style and short import libraries at this moment. We need to do something for that. Except that, supporting dlltool-style libraries should be straightforward.

 

This is a very unique scenario and I don't know of any tool that generates both styles of libraries.

I can create one if we need it for a test case but I think this would be overkill :) 

This is not actually very unique but rather common because you usually always want to link against kernel32.lib which contains short import libraries.

[Martell Malone]

If it's not urgent, I can take a look after finishing up x86 COFF port and fixing up remaining x64 bugs.

I will create a test case for it then :)

 This is not actually very unique but rather common because you usually always want to link against kernel32.lib which contains short import libraries.

Originally I thought you meant having both import types in the same lib. 

You actually meant 2 separate libs one with each type. :)

I would note however that mingw doesn't use the kernel32.lib provided by MS

It has def file that was originally generated from the dll with the gendef tool which is then given to dlltool when building the mingw crt.

If it is a mingw lib being used with visual studio however then yes this needs to be supported

There are projects that have dll's built with mingw-w64 and that are linked into visual studio like the vlc media player windows 8 app