[llvm-dev] RFC: Revisiting LLD-as-a-library design

[Reid Kleckner via llvm-dev]

Hey all,

Long ago, the LLD project contributors decided that they weren't going to design LLD as a library, which stands in opposition to the way that the rest of LLVM strives to be a reusable library. Part of the reasoning was that, at the time, LLD wasn't done yet, and the top priority was to finish making LLD a fast, useful, usable product. If sacrificing reusability helped LLD achieve its project goals, the contributors at the time felt that was the right tradeoff, and that carried the day.

However, it is now ${YEAR} 2021, and I think we ought to reconsider this design decision. LLD was a great success: it works, it is fast, it is simple, many users have adopted it, it has many ports (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to run the linker as a library, and they aren't satisfied with the option of launching a child process. Some users are interested in process reuse as a performance optimization, some are including the linker in the frontend. Who knows. I try not to pre-judge any of these efforts, I think we should do what we can to enable experimentation.

So, concretely, what could change? The main points of reusability are:

- Fatal errors and warnings exit the process without returning control to the caller

- Conflicts over global variables between threads

Error recovery is the big imposition here. To avoid a giant rewrite of all error handling code in LLD, I think we should *avoid* returning failure via the llvm::Error class or std::error_code. We should instead use an approach more like clang, where diagnostics are delivered to a diagnostic consumer on the side. The success of the link is determined by whether any errors were reported. Functions may return a simple success boolean in cases where higher level functions need to exit early. This has worked reasonably well for clang. The main failure mode here is that we miss an error check, and crash or report useless follow-on errors after an error that would normally have been fatal.

Another motivation for all of this is increasing the use of parallelism in LLD. Emitting errors in parallel from threads and then exiting the process is risky business. A new diagnostic context or consumer could make this more reliable. MLIR has this issue as well, and I believe they use this pattern. They use some kind of thread shard index to order the diagnostics, LLD could do the same.

Finally, we'd work to eliminate globals. I think this is mainly a small matter of programming (SMOP) and doesn't need much discussion, although the `make` template presents interesting challenges.

Thoughts? Tomatoes? Flowers? I apologize for the lack of context links to the original discussions. It takes more time than I have to dig those up.

Reid

[Petr Hosek via llvm-dev]

A big +1 from our side since we have a potential use case for LLD-as-a-library (I was going to write a similar RFC but you beat me to it).

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[David Blaikie via llvm-dev]

+1 generally.

Petr - if you have particular context for your use case, might be handy to have as reference/motivation.
Reid - if you have any particular use case of your own in mind, or links to other discussions/users who are having friction with the current state of affairs, would be hand to have.

(though I don't generally want the thread to become about picking apart those use cases - library-based design and flexibility is a fairly core tenant of the LLVM project overall regardless of the validity of specific use cases)

[Andrew Kelley via llvm-dev]

Throwing flowers on behalf of the Zig project 🌻🥀 :)

Currently Zig embeds LLD with a renamed main() and then exposes `zig
ld.lld`, `zig ld64.lld`, `zig lld-link`, and `zig wasm-ld` which call
into this renamed main() function. Then when Zig wants to invoke LLD, it
executes itself as a child process, using one of these sub-commands.

LLD-as-a-library would remove the need for this trick, improving
performance especially on systems such as Windows which have a high cost
of spawning child processes.

It also has the possibility to improve error reporting, avoiding the
need for parsing the stderr of LLD. Errors could be communicated in a
more semantic way, with less room for bugs.

With our upcoming self-hosted backend, compile times are on order of
milliseconds, and so child process spawning times in order to invoke LLD
end up being a substantial portion of total compilation time. That said,
we are also entering the linking space to provide an alternative to LLD,
but it will be some time before any possibility of removing a dependency
on LLD, so this RFC would be greatly beneficial to the Zig project.

Cheers,
Andrew

[Tom Stellard via llvm-dev]

On 6/10/21 11:14 AM, Reid Kleckner via llvm-dev wrote:

> Hey all,
>
> Long ago, the LLD project contributors decided that they weren't going to design LLD as a library, which stands in opposition to the way that the rest of LLVM strives to be a reusable library. Part of the reasoning was that, at the time, LLD wasn't done yet, and the top priority was to finish making LLD a fast, useful, usable product. If sacrificing reusability helped LLD achieve its project goals, the contributors at the time felt that was the right tradeoff, and that carried the day.
>

I think it would be great to move in this directions. It's a little bit unclear
at the moment whether or not the library use case is supported, because we
do include headers and libraries in the install targets.

As a package maintainer my wish list for an LLD library is:

1. Single shared object: https://reviews.llvm.org/D85278
2. All symbols are assumed private unless explicitly given library visibility.
3. Some subset of the API that is stable across major releases.

-Tom

> However, it is now ${YEAR} 2021, and I think we ought to reconsider this design decision. LLD was a great success: it works, it is fast, it is simple, many users have adopted it, it has many ports (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to run the linker as a library, and they aren't satisfied with the option of launching a child process. Some users are interested in process reuse as a performance optimization, some are including the linker in the frontend. Who knows. I try not to pre-judge any of these efforts, I think we should do what we can to enable experimentation.
>
> So, concretely, what could change? The main points of reusability are:
> - Fatal errors and warnings exit the process without returning control to the caller
> - Conflicts over global variables between threads
>
> Error recovery is the big imposition here. To avoid a giant rewrite of all error handling code in LLD, I think we should *avoid* returning failure via the llvm::Error class or std::error_code. We should instead use an approach more like clang, where diagnostics are delivered to a diagnostic consumer on the side. The success of the link is determined by whether any errors were reported. Functions may return a simple success boolean in cases where higher level functions need to exit early. This has worked reasonably well for clang. The main failure mode here is that we miss an error check, and crash or report useless follow-on errors after an error that would normally have been fatal.
>
> Another motivation for all of this is increasing the use of parallelism in LLD. Emitting errors in parallel from threads and then exiting the process is risky business. A new diagnostic context or consumer could make this more reliable. MLIR has this issue as well, and I believe they use this pattern. They use some kind of thread shard index to order the diagnostics, LLD could do the same.
>
> Finally, we'd work to eliminate globals. I think this is mainly a small matter of programming (SMOP) and doesn't need much discussion, although the `make` template presents interesting challenges.
>
> Thoughts? Tomatoes? Flowers? I apologize for the lack of context links to the original discussions. It takes more time than I have to dig those up.
>
> Reid
>

[David Blaikie via llvm-dev]

On Thu, Jun 10, 2021 at 3:16 PM Tom Stellard via llvm-dev <llvm...@lists.llvm.org> wrote:

On 6/10/21 11:14 AM, Reid Kleckner via llvm-dev wrote:
> Hey all,
>
> Long ago, the LLD project contributors decided that they weren't going to design LLD as a library, which stands in opposition to the way that the rest of LLVM strives to be a reusable library. Part of the reasoning was that, at the time, LLD wasn't done yet, and the top priority was to finish making LLD a fast, useful, usable product. If sacrificing reusability helped LLD achieve its project goals, the contributors at the time felt that was the right tradeoff, and that carried the day.
>

I think it would be great to move in this directions.  It's a little bit unclear
at the moment whether or not the library use case is supported, because we
do include headers and libraries in the install targets.

As a package maintainer my wish list for an LLD library is:

1. Single shared object: https://reviews.llvm.org/D85278
2. All symbols are assumed private unless explicitly given library visibility.

Is this ^ consistent with how other parts of LLVM are handled? My understanding was generally LLVM's API is wide/unbounded and not stable. I'd hesitate to restrict future libraries in some way due to some benefits that provides - ease of refactoring (LLVM's ability to be changed frequently is very valuable).
 

3. Some subset of the API that is stable across major releases.

A limited stable C API seems plausible to me, if there's need.

- Dave
 

[Tom Stellard via llvm-dev]

On 6/10/21 3:28 PM, David Blaikie wrote:

> On Thu, Jun 10, 2021 at 3:16 PM Tom Stellard via llvm-dev <llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>> wrote:
>
> On 6/10/21 11:14 AM, Reid Kleckner via llvm-dev wrote:
> > Hey all,
> >
> > Long ago, the LLD project contributors decided that they weren't going to design LLD as a library, which stands in opposition to the way that the rest of LLVM strives to be a reusable library. Part of the reasoning was that, at the time, LLD wasn't done yet, and the top priority was to finish making LLD a fast, useful, usable product. If sacrificing reusability helped LLD achieve its project goals, the contributors at the time felt that was the right tradeoff, and that carried the day.
> >
>
> I think it would be great to move in this directions.  It's a little bit unclear
> at the moment whether or not the library use case is supported, because we
> do include headers and libraries in the install targets.
>
> As a package maintainer my wish list for an LLD library is:
>

> 1. Single shared object: https://reviews.llvm.org/D85278 <https://reviews.llvm.org/D85278>

> 2. All symbols are assumed private unless explicitly given library visibility.
>
>
> Is this ^ consistent with how other parts of LLVM are handled? My understanding was generally LLVM's API is wide/unbounded and not stable. I'd hesitate to restrict future libraries in some way due to some benefits that provides - ease of refactoring (LLVM's ability to be changed frequently is very valuable).
>

The single shared object is consistent with clang and llvm, but
not the private symbols by default. We have discussed changing
this in clang and llvm, though, and for a library like lld that is
smaller than the clang and llvm libraries, it seems like it would
be an easier task and something that would be useful to do right
from the start.

-Tom

> 3. Some subset of the API that is stable across major releases.
>
>
> A limited stable C API seems plausible to me, if there's need.
>
> - Dave
>
>
> -Tom
>
>
> > However, it is now ${YEAR} 2021, and I think we ought to reconsider this design decision. LLD was a great success: it works, it is fast, it is simple, many users have adopted it, it has many ports (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to run the linker as a library, and they aren't satisfied with the option of launching a child process. Some users are interested in process reuse as a performance optimization, some are including the linker in the frontend. Who knows. I try not to pre-judge any of these efforts, I think we should do what we can to enable experimentation.
> >
> > So, concretely, what could change? The main points of reusability are:
> > - Fatal errors and warnings exit the process without returning control to the caller
> > - Conflicts over global variables between threads
> >
> > Error recovery is the big imposition here. To avoid a giant rewrite of all error handling code in LLD, I think we should *avoid* returning failure via the llvm::Error class or std::error_code. We should instead use an approach more like clang, where diagnostics are delivered to a diagnostic consumer on the side. The success of the link is determined by whether any errors were reported. Functions may return a simple success boolean in cases where higher level functions need to exit early. This has worked reasonably well for clang. The main failure mode here is that we miss an error check, and crash or report useless follow-on errors after an error that would normally have been fatal.
> >
> > Another motivation for all of this is increasing the use of parallelism in LLD. Emitting errors in parallel from threads and then exiting the process is risky business. A new diagnostic context or consumer could make this more reliable. MLIR has this issue as well, and I believe they use this pattern. They use some kind of thread shard index to order the diagnostics, LLD could do the same.
> >
> > Finally, we'd work to eliminate globals. I think this is mainly a small matter of programming (SMOP) and doesn't need much discussion, although the `make` template presents interesting challenges.
> >
> > Thoughts? Tomatoes? Flowers? I apologize for the lack of context links to the original discussions. It takes more time than I have to dig those up.
> >
> > Reid
> >
> > _______________________________________________
> > LLVM Developers mailing list

> > llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>

> >
>
> _______________________________________________
> LLVM Developers mailing list

> llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>

[Fāng-ruì Sòng via llvm-dev]

> thoughts? Tomatoes? Flowers? I apologize for the lack of context links to the original discussions. It takes more time than I have to dig those up.

+1

> - Fatal errors and warnings exit the process without returning control to the caller

This means every single fatal() call needs scrutiny. The function is
noreturn and there are 147 references.
In many places returning from fatal() can indeed crash.

> Another motivation for all of this is increasing the use of parallelism in LLD. Emitting errors in parallel from threads and then exiting the process is risky business. A new diagnostic context or consumer could make this more reliable. MLIR has this issue as well, and I believe they use this pattern. They use some kind of thread shard index to order the diagnostics, LLD could do the same.

Yes, I remember that I refrained from warn() in some parallel*,
because warn() becomes error() in --fatal-warnings mode and error() is
similar to fatal() after --error-limit is reached.

On Thu, Jun 10, 2021 at 8:57 PM Tom Stellard <tste...@redhat.com> wrote:
>
> On 6/10/21 3:28 PM, David Blaikie wrote:
> > On Thu, Jun 10, 2021 at 3:16 PM Tom Stellard via llvm-dev <llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>> wrote:
> >
> > On 6/10/21 11:14 AM, Reid Kleckner via llvm-dev wrote:
> > > Hey all,
> > >
> > > Long ago, the LLD project contributors decided that they weren't going to design LLD as a library, which stands in opposition to the way that the rest of LLVM strives to be a reusable library. Part of the reasoning was that, at the time, LLD wasn't done yet, and the top priority was to finish making LLD a fast, useful, usable product. If sacrificing reusability helped LLD achieve its project goals, the contributors at the time felt that was the right tradeoff, and that carried the day.
> > >
> >
> > I think it would be great to move in this directions. It's a little bit unclear
> > at the moment whether or not the library use case is supported, because we
> > do include headers and libraries in the install targets.
> >
> > As a package maintainer my wish list for an LLD library is:
> >
> > 1. Single shared object: https://reviews.llvm.org/D85278 <https://reviews.llvm.org/D85278>
> > 2. All symbols are assumed private unless explicitly given library visibility.
> >
> >
> > Is this ^ consistent with how other parts of LLVM are handled? My understanding was generally LLVM's API is wide/unbounded and not stable. I'd hesitate to restrict future libraries in some way due to some benefits that provides - ease of refactoring (LLVM's ability to be changed frequently is very valuable).
> >
>
> The single shared object is consistent with clang and llvm, but
> not the private symbols by default. We have discussed changing
> this in clang and llvm, though, and for a library like lld that is
> smaller than the clang and llvm libraries, it seems like it would
> be an easier task and something that would be useful to do right
> from the start.

I support that we compile lld source files with -fvisibility=hidden on
ELF systems.

lld::*::link may be the only API which need LLVM_EXTERNAL_VISIBILITY
(i.e. default visibility for ELF)
(LLVM_EXTERNAL_VISIBILITY is defined in llvm/include/llvm/Support/Compiler.h
Windows doesn't customize it currently.)

> -Tom
>
> > 3. Some subset of the API that is stable across major releases.
> >
> >
> > A limited stable C API seems plausible to me, if there's need.
> >
> > - Dave
> >
> >
> > -Tom
> >
> >
> > > However, it is now ${YEAR} 2021, and I think we ought to reconsider this design decision. LLD was a great success: it works, it is fast, it is simple, many users have adopted it, it has many ports (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to run the linker as a library, and they aren't satisfied with the option of launching a child process. Some users are interested in process reuse as a performance optimization, some are including the linker in the frontend. Who knows. I try not to pre-judge any of these efforts, I think we should do what we can to enable experimentation.
> > >
> > > So, concretely, what could change? The main points of reusability are:
> > > - Fatal errors and warnings exit the process without returning control to the caller
> > > - Conflicts over global variables between threads
> > >
> > > Error recovery is the big imposition here. To avoid a giant rewrite of all error handling code in LLD, I think we should *avoid* returning failure via the llvm::Error class or std::error_code. We should instead use an approach more like clang, where diagnostics are delivered to a diagnostic consumer on the side. The success of the link is determined by whether any errors were reported. Functions may return a simple success boolean in cases where higher level functions need to exit early. This has worked reasonably well for clang. The main failure mode here is that we miss an error check, and crash or report useless follow-on errors after an error that would normally have been fatal.
> > >
> > > Another motivation for all of this is increasing the use of parallelism in LLD. Emitting errors in parallel from threads and then exiting the process is risky business. A new diagnostic context or consumer could make this more reliable. MLIR has this issue as well, and I believe they use this pattern. They use some kind of thread shard index to order the diagnostics, LLD could do the same.
> > >
> > > Finally, we'd work to eliminate globals. I think this is mainly a small matter of programming (SMOP) and doesn't need much discussion, although the `make` template presents interesting challenges.
> > >
> > > Thoughts? Tomatoes? Flowers? I apologize for the lack of context links to the original discussions. It takes more time than I have to dig those up.
> > >
> > > Reid
> > >
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>
> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> > >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm...@lists.llvm.org <mailto:llvm...@lists.llvm.org>
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >
>

--
宋方睿

[Renato Golin via llvm-dev]

Bug +1 from me, too. 

The reasons of the past were clear, and they helped make the project a success. Kudos to all involved.

The current reasons are also clear, especially error handling and reuse of infrastructure.

Cheers,

Renato 

[Petr Hosek via llvm-dev]

Sure, I'm thinking about porting LLVM to Fuchsia and our process creation is not (and might never be) as fast as on *NIX, it's actually closer to Windows, so I'm interested in integrating LLD into Clang to reduce the cost (similarly to what Zig does).

More generally, I'd also like to explore alternative models like llvm-buildozer or a client-server model similar to what Daniel described in his talk, and having LLD-as-a-library makes that easier.

[James Henderson via llvm-dev]

No objections here (although I don't have a specific use-case currently).

Regarding the error handling, I support some sort of callback approach to report the errors (https://www.youtube.com/watch?v=YSEY4pg1YB0). This doesn't solve the problem of what to do after a fatal error has been reported. In the debug line parsing code which inspired that talk, we had a concept of unrecoverable and recoverable errors, whereby the parser would either stop parsing if it found something it couldn't recover from, by bailing out of the function, or it would set some assumed values and continue parsing. This may work for some cases in LLD, but the fatal cases need to stop the linking completely, so we'll need some way to bail out of the LLD call stack in those cases still somehow - personally, I think we should use llvm::Error for that up to the point of public interface with the library, to avoid the failure being unchecked. The error callbacks could then return Error to allow a client to force LLD to stop, even if the error would otherwise be non-fatal.

James

[Peter Smith via llvm-dev]

One of the earliest discussions about the LLD as a library design, at least after it had matured enough to be practical was this rather long thread https://lists.llvm.org/pipermail/llvm-dev/2016-December/107981.html

 

I don’t have any objections about making LLD more useable as a library.

 

What I would say is that we should come up with a good idea of what the functionality needed from the library is. For example I can see one use case with a relatively coarse interface that is similar to running LLD from the command line, with object files passed in memory buffers. I can see that working as an extension of the existing design. A more ambitious use case would permit more fine grain control of the pipeline, or construct LLD data structures directly rather than using object files could require quite a bit more work. I think people are thinking along the lines of the former, but it is worth making sure.

 

I think one of the reasons the library use case faltered was that there was no-one with a use case that was able to spend enough time to make it happen. The existing maintainers had enough work to do to catch up with Gold and BFD. Do we have someone willing to do the work?

 

Peter

[Jean-Daniel via llvm-dev]

One other point to consider for library usage is memory management.

I don’t know the state of LLD  today, but I think it was purposefully leaking memory as a tradeoff for performance at some point in the past (why deallocating everything before exiting main as the system will take care of this).

This presentation even says: 

What makes LLD so fast? - Previous FOSDEM Editionshttps://archive.fosdem.org › llvm_lld › slides › W…

“ • Large amounts of memory allocated up front and never released. »

[Petr Hosek via llvm-dev]

That can be controlled, see https://github.com/llvm/llvm-project/blob/a845dc1e562c20db54018a121eb01970e76602db/lld/ELF/Driver.cpp#L127

[David Blaikie via llvm-dev]

On Fri, Jun 11, 2021 at 3:56 AM Peter Smith via llvm-dev <llvm...@lists.llvm.org> wrote:

One of the earliest discussions about the LLD as a library design, at least after it had matured enough to be practical was this rather long thread https://lists.llvm.org/pipermail/llvm-dev/2016-December/107981.html

 

I don’t have any objections about making LLD more useable as a library.

 

What I would say is that we should come up with a good idea of what the functionality needed from the library is. For example I can see one use case with a relatively coarse interface that is similar to running LLD from the command line, with object files passed in memory buffers. I can see that working as an extension of the existing design. A more ambitious use case would permit more fine grain control of the pipeline, or construct LLD data structures directly rather than using object files could require quite a bit more work. I think people are thinking along the lines of the former, but it is worth making sure.

Not sure it's super important to decide much of that up-front. Once the general philosophy is agreed upon and the most fundamental issues are addressed for even the narrowest/simplest library use - other things can be done as enhancements beyond that.

A stretch/eventual goal could be to share some code between lld and LLVM's ORC JITLink - so that JIT and AOT linking share more code/diverge less.
 

[Jez Ng via llvm-dev]

As one of the people working on the new Mach-O backend, my main concerns are:

1. The Mach-O backend is still very much in flux, and will likely remain so until the end of this year. Whoever undertakes this library-fication should sync up with us to avoid e.g. awkward merge conflicts.
2. Performance is very important to us, and this library-fication should not regress it. Right now, we don't have a good benchmarking service set up (the LNT server only benchmarks LLD-ELF); we've been mostly just profiling things locally. I can send these benchmarking instructions to whomever takes on this effort.
   

Jez

[Michael Spencer via llvm-dev]

Adding Erik (not subscribed) who has previously had issues with LLD not being a library to provide some additional use cases.

- Michael Spencer

[pawel k. via llvm-dev]

Hello all,

Do I understand correctly lld is binary linker? Does it take arch/os specific or llvm-specific object files as input or both?

Im very young devel llvm- and clang-wise albeit if libraryfication of lld doesnt involve too heavy algorithmical and architectural changes and someone can intro me on working with git (which branch to start from on which branch to work whether to create separate branch and how), i could try to prepare working prototype of this solution. Please note though im still learning.

For macho code, we could use samples in testsuite where I could make sure nothing breaks after my changes. We could use some such samples too for lto. In perfect world, we could use some testset too for each arch/os combo.

As on avoiding merge conflicts with current devel including macho code, id suggest finnishing work there and stabilizing lld first, then when no new code is changed heavily or introduced anymore, i could try to focus on architecture rework safely.

As on unification with jit or lto as was mentioned, i dont know enough about this task yet.

As on avoiding/fixing memory leaks between compilation units etc, i think i could try to fix those.

Please note though I still dont have clear view as on how to make lld process more parallel so I may be not yet usable for this project.

Best regards,

Pawel

[Shoaib Meenai via llvm-dev]

To second what Jez said here: the new Mach-O backend is still pretty young (e.g. it wasn’t even the default Mach-O backend in LLVM 12). It’s being actively developed, and there’s still the possibility of fairly invasive changes being required to support new features or as we gain more implementation experience.

 

One of our main motivations for LLD for Mach-O was to improve incremental build speeds for our developers. We’ve found that link speed is a major component of incremental build times; LLD’s speed is a huge win there, and we care a lot about maintaining that speed. I’m CCing Nico, since he’s also been actively benchmarking LLD for Mach-O against Chromium builds (and reporting and fixing speed regressions).

 

Reid’s proposals (better error handling and eliminating globals) are completely reasonable. In general though, I really appreciate the LLD codebase’s simplicity and think it’s very valuable for understanding and maintaining the code. I haven’t worked too much with the rest of LLVM or Clang, so I’m not at all trying to compare LLD with them or cast aspersions on those codebases; I’m just speaking from my personal perspective. For example, having (mostly) separate codebases for each LLD port set off my “code duplication” spidey senses when I first started working with LLD, but while it does lead to some amount of duplication, there’s also a lot of subtle behavior differences between platforms, and having some amount of duplication is IMO a better tradeoff than e.g. having common functions that have a bunch of conditionals for each platform, or trying to come up with common abstract interfaces that are specialized for each platform, or so on. I really hope we can maintain that simplicity to whatever extent possible.

 

Thanks,

Shoaib

 

From: llvm-dev <llvm-dev...@lists.llvm.org> on behalf of Jez Ng via llvm-dev <llvm...@lists.llvm.org>
Reply-To: Jez Ng <je...@fb.com>
Date: Friday, June 11, 2021 at 2:14 PM
To: "llvm...@lists.llvm.org" <llvm...@lists.llvm.org>
Subject: Re: [llvm-dev] RFC: Revisiting LLD-as-a-library design

 

As one of the people working on the new Mach-O backend, my main concerns are:

[pawel k. via llvm-dev]

Agree here duplication is less of problem than gnuish/fsfish/binutilish/gccish spaghetti code with no abstractions etc. and thousands of conditionals etc.

- pawel

_______________________________________________

[Erik McClure via llvm-dev]

I use LLVM to compile WebAssembly to native code. The primary use-case for this is compiling WASM plugins for games - this is what Microsoft Flight Simulator 2020 uses it for. Using the system linker is not an option on Windows, which does not ship link.exe by default, making LLD a mandatory requirement if you are using LLVM in any kind of end-user plugin scenario, as the average user has not installed Visual Studio.

This puts users of LLVM's library capabilities on windows in an awkward position, because in order to use LLVM as a library when compiling a plugin, one must use LLD, which cannot be used as a library. My current solution is to use LLD as a library anyway and maintain a fork of LLVM with the various global cleanup bugs patched (most of which have now made it into stable), along with a helper function that allows me to use LLD to read out the symbols of a given shared library (which is used to perform link-time validation of webassembly modules, because LLD makes it difficult to access any errors that happen).

If LLD wanted to become an actual library, I think it would need a better method of reporting errors than simply an stdout and stderr stream, although I don't know what this would look like. It would also be nice for it to expose the different link stages like LLVM does so that the application has a bit more control over what's going on. However, I don't really have any concrete ideas about what LLD should look like as a library, only that I would like it to stop crashing when I attempt to use it as one.

--

Sincerely, Erik McClure

[David Blaikie via llvm-dev]

Is this a JIT use case? Perhaps ORC would be applicable there.

Or is the intent to make on-disk linked shared libraries so they can be cached over multiple executions/etc, perhaps?

[Alexandre Ganea via llvm-dev]

Hello,

 

(David Blaikie)

> Reid - if you have any particular use case of your own in mind, or links to other discussions/users who are having friction with the current state of affairs, would be hand to have.

The topic came up last Thursday in the Windows call, please see notes in https://docs.google.com/document/d/1A-W0Sas_oHWTEl_x_djZYoRtzAdTONMW_6l1BH9G6Bo/

 

Speaking for Ubisoft, there’s a short term practical usage for us in llvm-buildozer. Neil Henning from Unity 3D raised a similar need for the Burst compiler.

 

Since I’ve already went through these topics before, I had a list of practical things to achieve the LLD-as-a-lib goal:

 

1. The cl::opt data needs to live in a stack context, LLVMContext maybe.
   
   
1. One pain point is usages of cl::location.
   
   Folks use that kind of pattern so that the global state can be referenced from other TUs, without having to reference the cl::opt global directly.
   For example:
   
   bool polly::ModelReadOnlyScalars;
   
   static cl::opt<bool, true> XModelReadOnlyScalars(
       "polly-analyze-read-only-scalars",
       cl::desc("Model read-only scalar values in the scop description"),
       cl::location(ModelReadOnlyScalars), cl::Hidden, cl::ZeroOrMore,
       cl::init(true), cl::cat(PollyCategory));
   
   
2. Make the CommandLineParser’s cl::opt_storage data live in a stack- or heap-based context.
   
   This is similar to point a. above, except that this applies to the implicit cl::opt state (when cl::location is omitted).
   There’s a PoC in https://reviews.llvm.org/D86351, lib/Support/CommandLine.cpp, L556
   
   
   

The rationale here is to have the ability to call LLD-as-a-lib (or Clang-as-a-lib for instance, or any other LLVM tool) in the same way as we do on the command-line. Essentially calling into LLD main() but in-process. Like mentioned in https://reviews.llvm.org/D86351 one of our objectives is to pass a CDB .json to a tool (llvm-buildozer) and build in-process.

2. The targets registry takes some time to initialize, it wouldn’t be desirable to do it every time we call into LLD-as-a-lib.
   
   With a few modifications the initialization can be made thread-safe, which is step towards making the LLD-as-a-lib entry point thread-safe.
   See PoC in https://reviews.llvm.org/D86351, lib/Support/TargetRegistry.cpp
   
   
3. Move LLD global variables into (LLVM? LLD?) context
   
   A first inception of this can be achieved manually, as in https://reviews.llvm.org/D86351, lld/COFF/*, see changes marked LLVM_THREAD_LOCAL, which obviously just a PoC. A better implementation would move these variables into a “LLD context”.
   A more advanced version would automatize this somehow, at least warn developers that a global was introduced.
   
   
4. How do we handle memory allocations?
   
   It seems the most sensible thing to do on the short term (when using LLD-as-a-lib) is to run with exitEarly = false , ie: https://github.com/llvm/llvm-project/blob/d480f968ad8b56d3ee4a6b6df5532d485b0ad01e/lld/include/lld/Common/ErrorHandler.h#L101
   
   Later, a smarter way to get full performance would be to put all allocations into a BumpPtrAllocator-like for each LLD-as-a-lib call -- which would have the effect of exitEarly = true.
   
   
5. Externalizing ThreadPool
   
   One thing that wasn’t mentioned in Reid’s initial post, is that we won’t be able to spawn ThreadPools anymore inside LLD, at least when used as-a-lib. For example, if we call several instances of LLD-as-a-lib, from multiple client threads, a ThreadPool needs to be provided externally, by the client application. Jobs spawned internally by each LLD instance would be queued on the client ThreadPool.
   I don’t think this should be very hard to do, but a first iteration could (temporarily) disable multi-threading in LLD when calling it as-a-lib.
   
   
6. Returning Errors
   
   I didn’t have any issues with that in the COFF driver, but the ELF driver for example has at least one place which throws a fatal() in a constructor. This leaves a half-initialized (pooled) object on the heap, which is later destructed when destructing the SpecificAlloc<>, which in turn can corrupt the heap. See description in https://reviews.llvm.org/rG45b8a741fbbf271e0fb71294cb7cdce3ad4b9bf3
   This is the primary reason why “canRunAgain” exists along with “safeLldMain”, https://github.com/llvm/llvm-project/blob/03769d9308fee79aa97149561bdbb6e3263789bd/lld/tools/lld/lld.cpp#L185
   
   I suppose we would need to come up with a way to bubble up errors in the driver, and only return them later when the LLD-as-a-lib call completes.
   
   
7. Implicit Windows Kernel states
   
   One difficulty is that Windows stores a implicit CWD (current working directory) state for each process. When issuing Win32 API calls with relative paths, the NT Kernel would concatenate that with the internal CWD. Essentially that means we cannot pass relative paths anymore to Win32 APIs, all the paths have to be made absolute prior, and we have to store the CWD per “context”, per LLD-as-a-lib call.
   This isn’t terribly complicated, but requires some piping, see https://reviews.llvm.org/D86351, changes in llvm/lib/Support/Windows/Path.inc
   There could be additional issues like this, and on Linux as well.
   
   
8. Splitting the LLD pipeline
   
   As mentioned in this thread, we could later have a C API to provide more granularity on the LLD pipeline. But on the short term I would leave that aside, until we have a working example with just one single LLD-as-a-lib entry point (that does the same as what lldmain() does today).
   
   

While we’re here, we had some other adjacent objectives with this work:

 

9. Being able to call Clang driver-as-a-lib in the same way as LLD
   
   Again, the objective here was to have the strictly same behavior when calling Clang-as-a-lib, as when calling clang-cl on the command-line.
   
   
10. Cache file accesses & stat
    
    We’ve seen some contention in the Windows kernel when accessing files. Being able to build in-process opens the door to sharing state between threads build different TUs, in the same way that clang-scan-deps does. There seems to be a FileSystemStatCache but it isn’t really used, anybody knows why? We could upstream a caching implementation for clang-scan-deps that could favor any other tools that do multithread in-process building.

 

11. LLD-as-a-DLL
    
    

One point that was raised recently is, being able to compile LLVM components as DLLs on Windows. This is all adjacent to LLD-as-a-lib, perhaps it isn’t desirable to always link statically LLD into the user’s application.

Does all this sound sensible? It would be nice to split the work between us, if possible. On the short term (next few weeks) I can work on 1. and 2.

 

Best,

Alex.

 

De : Reid Kleckner <r...@google.com>
Envoyé : June 10, 2021 2:15 PM
À : llvm-dev <llvm...@lists.llvm.org>; Fangrui Song <mas...@google.com>; Sam Clegg <s...@chromium.org>; Shoaib Meenai <sme...@fb.com>; g...@fb.com; je...@fb.com; Alexandre Ganea <alexand...@ubisoft.com>; Martin Storsjö <mar...@martin.st>
Objet : RFC: Revisiting LLD-as-a-library design

[Erik McClure via llvm-dev]

The point of using LLVM for compiling WASM is to take advantage of ahead-of-time optimizations that could cause hitches in a JIT. For example, it integrates polly to try to recover vectorization optimizations. The resulting DLL can then be cached and loaded instantly on every subsequent playthrough, without any possibility of hitching. Microsoft Flight Simulator 2020 also ships pre-compiled plugin DLLs on Xbox, which does not allow JITing code, but because these are compiled on developer machines the linker problem doesn't really apply in that situation. If they wanted to JIT webassembly, there are plenty of JIT runtimes to do that.

Regardless, I think it's kind of silly to say that instead of using a perfectly functional linker that LLVM has, someone should JIT the code. LLVM is a compiler backend - it should support using its own linker the same way people use LLVM, and if LLVM can be used as a library, then LLD should be usable as a library. Furthermore, there is no technical reason for LLD to not be a library. It's already almost all the way there, the maintainers simply don't bother testing to see when they forget to clean up one of the global caches.

--

Sincerely, Erik McClure

[David Blaikie via llvm-dev]

On Sat, Jun 12, 2021 at 10:54 AM Erik McClure <erik...@gmail.com> wrote:

The point of using LLVM for compiling WASM is to take advantage of ahead-of-time optimizations that could cause hitches in a JIT.

Curious what sort of hitches you're referring to - but probably far enough off-topic from this thread. (certainly the goal of the ORC JIT is to be as robust as AOT compilation, providing the same semantics, etc)
 

For example, it integrates polly to try to recover vectorization optimizations. The resulting DLL can then be cached and loaded instantly on every subsequent playthrough, 

Fair enough - that's what I was curious about, or whether there were some other circumstances/motivations for using LLD as a library (eg: perhaps some bugs/missing features of ORC that could be addressed, or lack of documentation/visibility/etc).
 

without any possibility of hitching. Microsoft Flight Simulator 2020 also ships pre-compiled plugin DLLs on Xbox, which does not allow JITing code, but because these are compiled on developer machines the linker problem doesn't really apply in that situation. If they wanted to JIT webassembly, there are plenty of JIT runtimes to do that.

Regardless, I think it's kind of silly to say that instead of using a perfectly functional linker that LLVM has, someone should JIT the code.

I didn't mean to suggest that - but that it sounded pretty close to a JIT-like use case & was curious if there was some non-fundamental blocker that lead to the use of LLD rather than ORC for what appeared like it might be a JIT-like use case.
 

LLVM is a compiler backend - it should support using its own linker the same way people use LLVM, and if LLVM can be used as a library, then LLD should be usable as a library. Furthermore, there is no technical reason for LLD to not be a library. It's already almost all the way there, the maintainers simply don't bother testing to see when they forget to clean up one of the global caches.

I don't disagree that LLD, like the rest of LLVM, would benefit from having a library-centric design.

- Dave
 

[Neil Henning via llvm-dev]

+1 on this proposal from Reid (thanks for bringing this to the list!)

I'll drop a brain dump of why we (Unity) would like this proposal: we're running into the same problems that Andrew noted for Zig above. At present we are using LLD to do an on-disk JIT effectively:

• this gets us debugging support for 'free' (write a PDB to disk next to the DLL, et voila you can debug)
• it lets us cache binaries that don't change across source file changes to reduce overall build times

We've seen cases where we call LLD as-a-subprocess 2000 times with this approach. While I noted on the Windows/COFF call that we are taking steps to try and mitigate the number of calls to LLD, we'll still likely be in the 100~ ish DLLs built and thus 100~ ish calls to LLD. The cost of spawning all these LLD subprocesses is a good 15% of our build pipeline. As an experiment I tried running LLD-as-a-library and serialized the accesses to the linker, and it was only a little bit slower than having N threads launching N instances of LLD. That gives me good hope that having an actual thread-safe way to run LLD will substantially reduce our build times, meaning happy users.

-Neil.

--

Neil Henning

Senior Software Engineer Compiler

unity.com

[pawel k. via llvm-dev]

Hello,

Im trying to gather complete data about each use case of this new libified lld.

Can I ask more details on difference between workflow/scenario with 2000 spawns of lld process and 100ish spawns/runs in threads for linking 100ish DLLs?

Is same number of binaries built in each scenario?

Where do 2000 spawns of lld come from? Are there 2000 binaries built or do You run in modifycode-recompile-relink same binary in succession workflow?

Please bear with me. Im still learning. I would like to understand all details of requirements for all subtasks of this task to try to fix all issues mentioned.

I got one more concern with "running lld in threads" which is requiring thread safety between runs. Is some data shared between those threads or could we in worst scenario duplicate it in each threads context?

Best regards,

Pawel Kunio

[Neil Henning via llvm-dev]

Can I ask more details on difference between workflow/scenario with 2000 spawns of lld process and 100ish spawns/runs in threads for linking 100ish DLLs?

Is same number of binaries built in each scenario?

For our in-the-Unity-editor version (which is basically an on-disk-JIT), for each Burst entry-point (a job or function pointer that we've specially attributed in C#) we compile a single DLL at present. That means if the user has 2000 of these entry-points, we create 2000 DLLs. We plan to reduce this to produce a single DLL per assembly (C#'s rough approximation of a module), which would be around the 20-100 mark.

 

Where do 2000 spawns of lld come from? Are there 2000 binaries built or do You run in modifycode-recompile-relink same binary in succession workflow?

2000 is the number of entry-points required to execute the given game. If a user edits a single entry-point, we'll recompile a single DLL only.

 

I got one more concern with "running lld in threads" which is requiring thread safety between runs. Is some data shared between those threads or could we in worst scenario duplicate it in each threads context?

We'd need something akin to LLVM's context to keep the data separate per thread I think.

[Reid Kleckner via llvm-dev]

Thanks for all the responses! I read through all the emails to try to summarize the points that were brought up. I apologize if I missed yours.

I think the majority of responses were from LLD users, and they are clearly in favor of this. Maintainers seem open to change with some concerns about things like performance and merge conflicts.

I think the important thing is that we've established a direction. If you review LLD code, consider reviewing and accepting patches that move in this direction. I will try to put together a patch for COFF and we can discuss the details of the code structure there.

As for the shared library discussion, I think the refactoring of error handling and global state should come first. Then we can add a narrow API and a shared library around it. I think fine grained APIs and extension points should come later, if at all. I think it's important for LLD to stay focused on the performance of its primary use case, and I wouldn't want abstractions and extension points to get in the way of that. For these use cases, Orc may be more appropriate. In the long run, though, anything can happen. :)

[Nico Weber via llvm-dev]

+1 to the concerns here.

Not too surprisingly the concerns are from people who actually work on lld [1], who are likely to value hackability. I'd argue that hackability for people working on lld is ultimately beneficial for users of lld too though.

> I think the important thing is that we've established a direction. If you
> review LLD code, consider reviewing and accepting patches that move in this
> direction.

I'm not surprised that users of a software packet agree on having the people writing that software do more work. I don't think that necessarily counts as establishing a direction though.

1: ("1 year" chosen arbitrarily, but doesn't change much with "2 years")

$ git shortlog -nes lld --since '1 year ago' | head
   248 Jez Ng <je...@fb.com>
   167 Fangrui Song <i...@maskray.me>
   134 Nico Weber <tha...@chromium.org>
    69 Sam Clegg <s...@chromium.org>
    41 Greg McGary <g...@fb.com>
    29 Martin Storsjö <mar...@martin.st>
    26 Reid Kleckner <r...@google.com>
    22 Vy Nguyen <vy...@google.com>
    21 Petr Hosek <pho...@google.com>
    20 Georgii Rymar <gri...@accesssoftek.com>