[llvm-dev] [RFC] Building LLVM-Debuginfod
Noah Shutty via llvm-dev
Debuginfod is a simple protocol allowing a client to fetch debug information about a stripped binary from a server by supplying its build-id. We would like to add debuginfod support to LLVM to make it easier for users to debug stripped binaries.
In summary, we would like to:
Build an LLVM debuginfod client library in LLVM.
Integrate the client library into llvm-symbolizer and other tools.
Build a simple LLVM debuginfod server library for local development use.
The client and server tasks are separate and we have already begun work on the client side. The server is nothing more than a simple web server that serves static content associated with the build_id in the URL. We are eager to get feedback and advice from the community.
HTTP Client
The debuginfod client library will require an HTTP client, which should be fully featured to maximize compatibility across various network configurations and debuginfo server implementations. We were leaning towards libcurl which is widely available.
HTTP Server
The debuginfod server is essentially a static file server using HTTP over TCP/IP. We propose to use the TCPSocket library from lldb-server and build a very small HTTP file server on top using LLVM regular expressions. We built a proof-of-concept, more details can be found below under HTTP Server Benchmarks. This would require a refactor of the LLDB IOObject hierarchy into LLVM as described in our previous RFC.
As a short-term solution, since the refactor may take some time, we propose to use the lightweight cpp-httplib library to get a server up and running.
Database
Debuginfod needs to run key-value lookup queries to serve requests. Specifically, it must implement these logical mappings:
build_id -> executable_path
(build_id, source_path) -> source_path
In pursuit of a mechanism to perform these lookups, we can draw inspiration from clangd's indexing scheme and Apple’s index-while-building feature. Both of these implementations ultimately solve the same problem, supporting fast lookups of e.g. symbol occurrences in source code, but they take different approaches. Clangd produces indexes in memory and then writes them to disk in a way that enables efficient queries on-demand. Apple’s index-while-building libraries instead dump unindexed record and unit files to a directory using the LLVM bitstream file format. These files must then be indexed into lookup tables to answer queries on-demand – a task left to other tooling. For example, Apple’s IndexStoreDB tool ingests the record and unit files to produce key-value tables stored on-disk with LMDB.
To estimate the size of the database needed for a local debuginfo server, we indexed the binaries of the LLVM project (compiled with standard Debug mode options) using elfutils-debuginfod. The resulting SQLite database occupied just 35MB on disk, which could easily fit in a typical developer’s RAM. Of course, production debuginfo servers may index far more binaries, and the database could easily reach several GB.
Based on the small size required for a local debuginfod database, we think it should be fine to take the clangd-style approach out-of-the-box, while providing an interface compatible with LMDB (or LevelDB, etc.) to assist with production debuginfod servers. Specifically, when built without LMDB, llvm-debuginfod would index debuginfo using a few in-memory hash tables, then atomically serialize them to LLVM’s OnDiskHashTable (or subclasses). We would leave LMDB libraries as an optional dependency that could be used for production debuginfod servers.
Build system integration
llvm-debuginfod would be built as an implicit project similar to llvm-objdump and located at ./llvm/tools/llvm-debuginfod.
The client introduces a mandatory dependency on libcurl. This could be handled similarly to libxml in the llvm/CMakeLists.txt.
At first, the server will introduce a mandatory dependency on cpp-httplib until we can complete the refactor to move LLDB’s TCPSocket into LLVM. After the refactor the server will have no additional mandatory dependencies. The server will have optional dependencies on both cpp-httplib (or another HTTP server library) and LMDB. These optional dependencies would be intended for production debuginfod servers.
Client integration with other LLVM tools
The llvm-debuginfod server itself will require client functionality provided by a llvm-libdebuginfod-client library for federation to other debuginfo servers. We would wrap this library with a thin llvm-debuginfod-find tool in the same source directory with functionality similar to the debuginfod-find tool in elfutils. The library will make it simpler to bring the fetch capabilities to other tools like llvm-symbolizer in a nearly-uniform way. Specifically, the llvm-libdebuginfod-client library code will query the servers in the DEBUGINFOD_URLS environment variable for a given build_id. This client would be invoked from tools like llvm-symbolizer by adding a small (~10 line) change to the tool. The needed changes may be slightly different for each tool due to the different ways that debuginfo are found and handled. This makes it less of a “black-box” replacement than FUSE-based approaches, but leaves tool designers more flexibility to select if / when / how debuginfod servers are queried.
Implementation Schedule
We propose to split the implementation of llvm-debuginfod into 4 patches.
Patch 1 will add the HTTP client dependency (libcurl) to the build system.
Patch 2 will create the client library llvm-debuginfod-client and implement llvm-debuginfod-find. Public debuginfo servers and/or a local python http.server (f.k.a. SimpleHTTPServer) could be used for the initial test suite.
Patch 3 will make the changes to llvm-symbolizer needed for it to use the debuginfod-find library.
Patch 4 will implement debuginfod (the server).
These patches have the following logical dependencies:
Why another implementation of debuginfod?
Beyond the licensing aspect, there are benefits of multiple cross-compatible implementations of the debuginfo client and server., There is also an opportunity to optimize performance and add new features beyond elfutils’ debuginfod. For example, we could wrap the client with LLVM’s FS caching, fix issues such as infinite traversal when following symlinks, and extend support to Mach-O and other binary formats.
HTTP Server Benchmarks
The debuginfod HTTP server is essentially a static file server. We expect that users who run production debuginfo servers will implement some advanced / customized configuration with load balancing etc. We wish to target developers who will want something simple that works “out of the box” on their local machine. With this user in mind, we should minimize dependencies and maintain good perf for even large (say, 5GB) debuginfo.
To help pick an HTTP server, I have run preliminary tests comparing the following:
“llvm-httpd”, explained below.
These servers were chosen to provide representatives from across the spectrum of candidate libraries. nginx and h2o are highly-optimized and fully-featured configurable standalone webservers. libmicrohttpd is a GNU library used by elfutils-debuginfod. cpp-httplib is a small single-header client-server library. Finally, “llvm-httpd” is a custom barebones 100-line HTTP fileserver using the posix socket and extended regex interfaces. This is intended as a stand-in for a small custom solution built on the LLDB TCPSocket library and LLVM regex library.
In the benchmarking test, a single 5.1GB randomly generated file is fetched using curl sequentially 100 times from localhost. h2o was configured to serve static files from a directory. cpp-httplib was configured with a custom file streaming response handler that I wrote because the library’s default static file serving code is not optimized for large files.
As you can see from the table above, the custom ‘llvm-httpd’ is only 33% slower on average than the fastest server, libmicrohttpd.
Based on this outcome, we propose to proceed as follows:
Move the LLDB IOObject hierarchy code to LLVM itself as explained in the previous RFC.
We note that Clangd remote is being built on gRPC, which provides a TCP server and HTTP 2.0 client / server library. However we do not plan to use the gRPC protocol so we were thinking it would be better to use the lightweight TCPSocket implementation which is already in the LLDB codebase.
Implement llvm-debuginfod using cpp-httplib at first, then switch to the LLDB TCPSocket code after the refactor to eliminate cpp-httplib dependency.
Leave the interface sufficiently modular to enable swapping out with a more featureful HTTP server library if needed.
Frank Ch. Eigler via llvm-dev
As a developer of elfutils/debuginfod, I read with interest your
intent to build an llvm reimplementation of the debuginfod stack.
Best of luck, enjoy!
I'm curious whether there were any indications that the existing code
base couldn't be used due to problems of some sort. AIUI, licensing
compatibility with LLVM is moot for out-of-process binaries like the
debuginfod server and the debuginfod-find client. -L symlink loops
are a SMOP. Was mach-o support the only real showstopper?
- FChE
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev