[LLVMdev] Linking tools

[Russell Wallace]

I'm trying to figure out exactly what the function and status of the different linking tools is. The impression I get is:

1. For linking multiple bitcode (either binary or text format) files together, llvm-link is the current and future intended tool.

2. For converting bitcode files into (machine code) object files, llc is the current and future intended tool.

3. For linking multiple object files into an executable, llvm has hitherto relied on the system linker, but lld is the future intended tool.

Is any of this inaccurate or incomplete?

[Eric Christopher]

On Fri, Jul 10, 2015 at 3:37 PM Russell Wallace <russell...@gmail.com> wrote:

I'm trying to figure out exactly what the function and status of the different linking tools is. The impression I get is:

1. For linking multiple bitcode (either binary or text format) files together, llvm-link is the current and future intended tool.

2. For converting bitcode files into (machine code) object files, llc is the current and future intended tool.

llvm-link and llc are developer tools only. clang is designed to be the interface here (or your language specific driver).

 

3. For linking multiple object files into an executable, llvm has hitherto relied on the system linker, but lld is the future intended tool.

This is correct.

-eric

 

Is any of this inaccurate or incomplete?

_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[Russell Wallace]

On Fri, Jul 10, 2015 at 11:39 PM, Eric Christopher <echr...@gmail.com> wrote:

On Fri, Jul 10, 2015 at 3:37 PM Russell Wallace <russell...@gmail.com> wrote:

I'm trying to figure out exactly what the function and status of the different linking tools is. The impression I get is:

1. For linking multiple bitcode (either binary or text format) files together, llvm-link is the current and future intended tool.

2. For converting bitcode files into (machine code) object files, llc is the current and future intended tool.

llvm-link and llc are developer tools only. clang is designed to be the interface here (or your language specific driver).

Okay so for linking bitcode files together, what's the intended command? That is, 'clang x.bc y.bc' will generate an executable, but for generating a single larger bitcode file? Adding -emit-llvm gives an error unless you also add -c, but that just rewrites the original small bitcode files?

[Reid Kleckner]

I think your original description of the situation is accurate. llvm-link will take multiple bitcode files and spit out a big ball o' bitcode, but that's usually not sufficient for LTO, which is the main use case that we want to support. From the perspective of LTO, we just want users to be able to add -flto to their compile and link lines, and make that produce a faster executable, without the user ever being aware of the bitcode.

If your use case (static analysis, maybe?) requires the intermediate bitcode, we don't yet have a nice way to get that from clang and maybe we should add one. Maybe -emit-llvm on a link line like you suggested, but that discards information about non-bitcode object files.

Anyway, for now, llvm-link will do the job, but it isn't really meant to be a user facing tool.

Hope that explains things. :)

[Russell Wallace]

On Mon, Jul 27, 2015 at 5:52 PM, Reid Kleckner <r...@google.com> wrote:

I think your original description of the situation is accurate. llvm-link will take multiple bitcode files and spit out a big ball o' bitcode, but that's usually not sufficient for LTO, which is the main use case that we want to support.

To be clear I understand you: the reason it's usually not sufficient is because most programs use build systems that don't really provide an opportunity for such a step; they assume the compiler only needs to be told about one source file at a time right up until machine code linking time?

 

From the perspective of LTO, we just want users to be able to add -flto to their compile and link lines, and make that produce a faster executable, without the user ever being aware of the bitcode.

Yes indeed. I understand there is work being done on achieving this by following the usual build procedure, but essentially disguising bitcode files as object files until link time? 

If your use case (static analysis, maybe?) requires the intermediate bitcode, we don't yet have a nice way to get that from clang and maybe we should add one. Maybe -emit-llvm on a link line like you suggested, but that discards information about non-bitcode object files.

Anyway, for now, llvm-link will do the job, but it isn't really meant to be a user facing tool.

Right, I'm looking at both whole-program optimisation and static analysis. But I suppose as you say, llvm-link should do the job for now. 

Hope that explains things. :)

It does, thanks! 

[Reid Kleckner]

On Mon, Jul 27, 2015 at 9:59 AM, Russell Wallace <russell...@gmail.com> wrote:

On Mon, Jul 27, 2015 at 5:52 PM, Reid Kleckner <r...@google.com> wrote:

I think your original description of the situation is accurate. llvm-link will take multiple bitcode files and spit out a big ball o' bitcode, but that's usually not sufficient for LTO, which is the main use case that we want to support.

To be clear I understand you: the reason it's usually not sufficient is because most programs use build systems that don't really provide an opportunity for such a step; they assume the compiler only needs to be told about one source file at a time right up until machine code linking time?

In practice, it's not sufficient because there are usually pre-compiled objects passed into the link step, and symbols from the bitcode are referenced from those object files. Getting the precise list of symbols that are actually referenced externally is a big part of the value of LTO.

[Steve King]

On Mon, Jul 27, 2015 at 9:52 AM, Reid Kleckner <r...@google.com> wrote:

> From the perspective of LTO, we just want users to be able
> to add -flto to their compile and link lines, and make that produce a faster
> executable, without the user ever being aware of the bitcode.

For targets without GNU binutils and gcc driver support, has this goal
been achieved? The few times I've tried, Clang's hard-coded
dependencies on host GNU tools block LTO and linked binaries in
general. For my target, manually running llvm-link and llc is the
only way to get LTO-like output, but otherwise works pretty well.

For this same reason, I get worried when I hear maintainers state that
llvm-link, llc, llvm-mc, etc are developer only tools. GNUless
targets use these tools for production code for lack of working
alternatives.

If there's been recent progress on removing GNU dependencies, I'm all ears.

Cheers,
-steve

[Russell Wallace]

On Mon, Jul 27, 2015 at 6:34 PM, Reid Kleckner <r...@google.com> wrote:

In practice, it's not sufficient because there are usually pre-compiled objects passed into the link step, and symbols from the bitcode are referenced from those object files. Getting the precise list of symbols that are actually referenced externally is a big part of the value of LTO.

Right, yes, that's a very good point.

[Reid Kleckner]

On Mon, Jul 27, 2015 at 10:39 AM, Steve King <st...@metrokings.com> wrote:

On Mon, Jul 27, 2015 at 9:52 AM, Reid Kleckner <r...@google.com> wrote:

> From the perspective of LTO, we just want users to be able
> to add -flto to their compile and link lines, and make that produce a faster
> executable, without the user ever being aware of the bitcode.

For targets without GNU binutils and gcc driver support, has this goal
been achieved?  The few times I've tried, Clang's hard-coded
dependencies on host GNU tools block LTO and linked binaries in
general.  For my target, manually running llvm-link and llc is the
only way to get LTO-like output, but otherwise works pretty well.

For this same reason, I get worried when I hear maintainers state that
llvm-link, llc, llvm-mc, etc are developer only tools.  GNUless
targets use these tools for production code for lack of working
alternatives.

If there's been recent progress on removing GNU dependencies, I'm all ears.

Basically, LTO for projects that have pre-compiled objects requires integration with a real static linker. Currently we use plugins to integrate with binutils linkers, Mac ld64, and some other closed-source linkers. To cut this dependency, we need a new linker, which is what LLD is intended to become.

[Steve King]

Thanks and yes, I eagerly await Clang's switch to lld. Clang also
calls the gcc driver though, at least to invoke collect2. Will lld
allow clang to remove all GNU dependencies?