I'm interested in whether you plan to have this integrated in lld as well.
As far as I understand, this is going to be the de-facto way of
shipping for Mach-O binaries (at least, the ones released by Apple).
Please correct me if I'm wrong.
I tried to self-host lld on El Capitan and it fails because lld
doesn't really know about TBD files.
This, unfortunately, makes the linker not really usable for modern Mac
OS releases.
>
> The functionality of the tool is currently limited to Mach-O object files,
> but that is not a technical limitation. In making the tool open source I
> hope others will be able to take advantage of it too and extend its
> functionality to other object file formats.
>
>
> I initially developed the project as a CLANG project, but that was mostly
> for practical reasons (out-of-tree development, separate repo, etc). For the
> curious ones I pushed the repo to github (https://github.com/ributzka/tapi).
>
> I imagine, for example, that the reading/writing of TBD files is something
> that would fit better into the LLVM sources, which makes it available to
> other libraries and tools (e.g. LLVMObject, llvm-nm, lld, ...).
>
> I created a small patch that integrates it with llvm-nm and LLVMObject. This
> patch is not complete and I will split it up into smaller patches for
> review. I am providing it as a reference to get the discussion started.
>
> Please let me know what you think and bikeshed away :)
>
> Thanks
>
> Cheers,
> Juergen
>
>
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm...@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
--
Davide
"There are no solved problems; there are only problems that are more
or less solved" -- Henri Poincare
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
cfe-dev mailing list
cfe...@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Beyond my general esteem for using export lists for hermetic builds, I'm
working on automated native and cross builds for Darwin, and it would be
a lot more convenient if this was part of LLVM.
So what happened to this? I searched TAPI on differential and didn't see
anything. Given the general need for something like a standardized
export list to feed a linker instead of the shared library itself, and
the utility of this specific format given its use by a major platform, I
think it definitely should be upstreamed.
John
[1]
https://github.com/joyent/illumos-joyent/blob/master/usr/src/lib/README.mapfiles
File size is one reason. A TBD file is typically one third the size of the corresponding stub library for a single architecture. Multiple architectures dramatically increase the TBD advantage: a new architecture in TBD may cost as little as a few bytes if all architectures export the same functions, but each new architecture in a stub library requires duplicating its entire contents.
--
Greg Parker gpa...@apple.com Runtime Wrangler
Yes. I hope this can be reason enough. Hobbyists could push for LLD
support for Mach-O besides Apple, and if LLD is to displace other
linkers this is a necessary component as you say. Better to upstream now
before the code diverges than more work later? Conversely if nothing
happens, I doubt libtapi would be a greater drag on the codebase than
the MachO LLD code, so whatever cost/benefit analysis exists for keeping
that around could also apply to this.
> On 04/09/2018 06:39 PM, Greg Parker via cfe-dev wrote:
>> On Apr 9, 2018, at 3:23 PM, James Y Knight via cfe-dev
<cfe...@lists.llvm.org> wrote:
>> I'm not really clear on the actual benefits of the TBD file, and why
Apple migrated to them in the first place. Shouldn't a dynamic library
containing only the relevant parts (e.g. the dynamic symbol table) be
roughly comparable in size?...
> File size is one reason...
For the record, other small benefits are
- The inclusion of the path to the actual library, which as far as I
know is not something that can be done with a stub library. This allows
easy absolute or relative (with R(UN)PATH) linking. Comparatively,
passing the right -rpath and -rpath link is manual and (in my opinion)
harder to understand and cumbersome, and also not a solution for
absolute linking. I work with Nixpkgs of NixOS, where absolute path
linking is frequently an objective as part of a general principle of
avoiding indirection.
- YAML. The option for line-oriented structure allows for easy diffing
with conventional line-based diffing tools, which is useful for
debugging compatability issues. (e.g. Why did my new version remove
symbols? Why did my security update change anything at all?). Of course
one can just objdump and diff, but that wouldn't happen automatically
with version control, for example.
John
> Regardless of any of that, given that TBD files _are_ an integral part of the apple platform, supporting them is certainly a necessity in order to have a working apple linker. So, if making LLD work for Apple/MachO is the justification for adding TBD support to LLVM, that seems self-evidently a reasonable thing to do. On the other hand, it looks like the LLD mach-o code is unmaintained and nobody seems to be much interested in it. And having code for reading TBD files in LLVM seems not terribly interesting, unless it is as part of a project to make the LLD MachO linker actually functional and supported.
Yes. I hope this can be reason enough. Hobbyists could push for LLD support for Mach-O besides Apple, and if LLD is to displace other linkers this is a necessary component as you say. Better to upstream now before the code diverges than more work later? Conversely if nothing happens, I doubt libtapi would be a greater drag on the codebase than the MachO LLD code, so whatever cost/benefit analysis exists for keeping that around could also apply to this.
Seems like there are a few of us interested in this then. I new around here and don't really know how decisions are made, so what's next? Just open a diff with the entire library??
John
That sounds great to me, thanks Jake. I'm not Jurgen either, of
course, but I'm happy to assist you if he is unavailable. I'm not
also not qualified to audit the license, but do note Apple
formally also released some code at
https://opensource.apple.com/tarballs/tapi/. If there's anything
else I can do to help, let me know.
Cheers,
John
Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI. Diffs can be automatically added to review processes to ensure that changes to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of binaries to reduce the size of an SDK
3) Stubs are producible from TBDs (or should be) which means stubs for linking can be produced even if we don't directly support them in LLD. This lets you ship the smaller TBD files in place of larger binaries and still link things without direct linker support (assuming you already ship a toolchain with your SDK or expect your users to have this tool)
Since stubs are producible from TBDs I don't really see a downside. I think we need both, I was going to propose a yaml based representation for ELF for the above reasons anyhow.
On Apr 11, 2018, at 11:52 AM, James Y Knight via cfe-dev <cfe...@lists.llvm.org> wrote:On Tue, Apr 10, 2018 at 5:33 PM Jake Ehrlich via llvm-dev <llvm...@lists.llvm.org> wrote:Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI. Diffs can be automatically added to review processes to ensure that changes to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of binaries to reduce the size of an SDKI'm still skeptical that this is significant.
3) Stubs are producible from TBDs (or should be) which means stubs for linking can be produced even if we don't directly support them in LLD. This lets you ship the smaller TBD files in place of larger binaries and still link things without direct linker support (assuming you already ship a toolchain with your SDK or expect your users to have this tool)
Since stubs are producible from TBDs I don't really see a downside. I think we need both, I was going to propose a yaml based representation for ELF for the above reasons anyhow.Yea, a tool which can produce a .so from a textual description is certainly much less concerning than adding linker support for a new textual description format. If it's an official linker-supported format, it'd be yet another format that potentially needs to be standardized across multiple linkers, and kept compatible for"ever", etc. I just don't think that seems worthwhile for ELF.OTOH, a standalone tool which can convert from a "full" shared-object to an interface shared-object would be _great_ to have. If that tool also has some auxiliary textual I/O format it supports, I guess that's fine, too. (We do have some existing yaml <-> ELF support, via the "obj2yaml" and "yaml2obj" tools.)I'd note that reproducing all the things that are required/used from an ELF shared object during linking -- symbol type, binding-type, visibility, version, alignment (!), .gnu.warning messages, various important "SHT_NOTE" sections, and whatever other things I've forgotten about, will need to be a _significantly_ different format than what Apple has as their "TBD" format. Apple's format also has a bunch of special cases in it to make it easier to use for their platform, but a rather less generic tool. E.g., symbols starting with "_OBJC_CLASS_$" are recorded in the "objc-classes" field with the prefix removed, instead of just recording it as-is.So, I'd also caution that while the project of "import apple's libtapi into LLVM for LLD/MachO" and "Make a scheme to do interface shared-libs for ELF" might seem superficially related, I'd be very surprised if that actually ended up being the case. I would really not expect it to share just about anything at all other than the concept of being a textual description for a library.
That sounds great to me, thanks Jake. I'm not Jurgen either, of course, but I'm happy to assist you if he is unavailable. I'm not also not qualified to audit the license, but do note Apple formally also released some code at https://opensource.apple.com/tarballs/tapi/. If there's anything else I can do to help, let me know.
Cheers,
John
On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
Ideally Jurgen would cut up the code on github, put up an initial diff for a minimal viable tool, and then we would review it and then continue to copy code from the github repo into llvm and review it. I'm also willing to do that if Jurgen doesn't want to at this point though. I'd like the OK from Jurgen on that and I'd also like the OK from someone that the license stuff is all good to go (I'm not sure who should check licence stuff).
Best,Jake
The code on opensource.apple.com is the minimal code needed for libtapi to read TBD files with ld64. The code I pushed to GitHub on the other side includes the full TAPI tool source code. If you check the code you will see that I already use the LLVM license for everything, because the goal has always been to contribute this back to the community.I attached an initial patch for libtapi and nm support to my original email. I also have some updated patches for that too, but somehow got derailed into "lets rewrite libobject" discussions.
On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <llvm...@lists.llvm.org> wrote:
That sounds great to me, thanks Jake. I'm not Jurgen either, of course, but I'm happy to assist you if he is unavailable. I'm not also not qualified to audit the license, but do note Apple formally also released some code at https://opensource.apple.com/tarballs/tapi/. If there's anything else I can do to help, let me know.
Cheers,
John
On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
Ideally Jurgen would cut up the code on github, put up an initial diff for a minimal viable tool, and then we would review it and then continue to copy code from the github repo into llvm and review it. I'm also willing to do that if Jurgen doesn't want to at this point though. I'd like the OK from Jurgen on that and I'd also like the OK from someone that the license stuff is all good to go (I'm not sure who should check licence stuff).
Best,Jake