Bitcoin Core's migration to the CMake buildsystem
529 views
Skip to first unread message
Cory Fields
Aug 7, 2024, 5:52:59 PMAug 7
to Bitcoin Development Mailing List
Hi All
I realize this is not a Bitcoin Core list, but I think this is relevant enough to many downstream developers to bring up here.
Hennadii Stepanov (hebasto) has been working hard for several years now to replace Bitcoin Core's Autotools-based buildsystem with a modern CMake one. Most of the developers who support the current buildsystem have been reviewing this work chunk-by-chunk in hebasto's staging branch since February of last year. As I've been helping with guidance and review of this work, hebasto has asked me to share a quick summary of our progress.
Anyone who's ever been around for a buildsystem migration in a large software project knows what a massive undertaking this is, as well as the frustrations that come along with it. This one is no exception. So before going further, I'd like to commend hebasto on his hard work and patience to get this far. Thanks, hebasto!
In the process, we've upstreamed dozens of bugfixes and modernizations for our various build-time and run-time dependencies. Thanks especially to Michael Ford (fanquake) for that effort. In the true open-source spirit, other users of those dependencies have received some free improvements from this work.
Now that it's gone through a long round of review by the buildsystem devs, we believe it's in good enough shape to merge into master in the upcoming weeks as planned, soon after feature-freeze for v28 and branch-off for v29.
We recognize that this will be a bumpy time for Bitcoin Core dev. It's simply impossible to avoid problems with a change like this. We believe that common workflows should be mostly uninterrupted, but those of you targeting esoteric platforms or build configs are more likely to run into some issues. So this would be a good time (before merge!) for anyone who builds Bitcoin Core to give the CMake branch a try and let us know about any trouble you've run into.
Please re-read that last paragraph. This _will_ be a painful process. Please try to be constructive/productive with your bug reports.
The current PR can be found here: https://github.com/bitcoin/bitcoin/pull/30454
The build-related documentation has been updated there. Please consider that documentation to be canonical, so if there's a question that's not answered (or info that's stale), please mention it in the form of a review on the pull request so that everyone can benefit.
FAQ:
Q: Noooooooooo!!!
A: That's not a question :). Maintaining Autotools is a huge burden and it's only getting worse over time as it's virtually (though not entirely) unmaintained. As the person who introduced and maintained the current buildsystem and maintained it for years (though others like fanquake and hebasto have been more active for the last few) I'm happy to see it go. She sure was a good ship.
Q: Why CMake? Why not [my favorite buildsystem]
A: Because CMake is a perfectly reasonable choice for a modern free and open-source buildsystem with substantial momentum. And also because hebasto stepped up to actually do the work.
Q: Will Autotools continue to be supported?
A: Not in master, no. We (the buildsystem devs) don't believe it's worth the pain/effort of maintaining parallel buildsystems for a project as large as Bitcoin Core. We plan to delete everything relating to Autotools within days of merging CMake. However, we have no intention of backporting CMake to previous branches, so we'll have to maintain it to a small degree there.
Q: When/how will this affect me as a user of Bitcoin Core.
A: If don't build Bitcoin Core from source, or if you don't know what that means, you shouldn't notice any change.
Q: When/how will this affect me as a developer who interacts with Bitcoin Core?
A: For builders of Core who are not Core devs, You'll probably encounter a new way of building Core when v29 is released, sometime mid next year. At that point, rather than "./configure && make", you'll be using "cmake -S . -B build && cmake --build build". With any luck, by then, most of the bugs will have been reported and fixed, and builds will just work.
Q: How do I configure the build? Where are the old options?
A: All relevant options have been ported over. See the docs for more details, but as a hint, the ccmake tool is very useful for listing build options.
Additionally, hebasto has been keeping up with a feature-parity table here: https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e
Q: Why is X tool or Y feature off by default?
A: We've decided to stick to the CMake way of doing things as much as possible. This way, a quick web search for "how to do Z with CMake" will probably turn up helpful and correct answers. Although, this paradigm switch is where we anticipate the most friction and frustration for users during the transition. While Autotools convention is to automatically find and enable features when possible (like enabling a feature if a library happens to exist on your system), the common CMake way is to require explicit opt-ins. This leads to fewer surprises as there's less magic happening out of sight, but may require a small upfront time investment to get back to the builds you're used to.
Q: Are all of CMake's cool features supported?
A: Some, not all. Focus has been on getting the initial port done correctly, with less effort going to fringe features and non-makefile generators. Generally speaking, single-config Makefiles have received the most attention. Ninja should just work. Multi-config builds may work, but we don't guarantee/support that just yet. MSVC should be in good shape as that's hebasto's platform of choice. XCode is known to work, but it's currently not very usable as it's unorganized and the build is full of irrelevant warnings. We'll clean those up in due time, but they're not high priority as we consider them new features as opposed to part of the initial port.
I realize this is not a Bitcoin Core list, but I think this is relevant enough to many downstream developers to bring up here.
Hennadii Stepanov (hebasto) has been working hard for several years now to replace Bitcoin Core's Autotools-based buildsystem with a modern CMake one. Most of the developers who support the current buildsystem have been reviewing this work chunk-by-chunk in hebasto's staging branch since February of last year. As I've been helping with guidance and review of this work, hebasto has asked me to share a quick summary of our progress.
Anyone who's ever been around for a buildsystem migration in a large software project knows what a massive undertaking this is, as well as the frustrations that come along with it. This one is no exception. So before going further, I'd like to commend hebasto on his hard work and patience to get this far. Thanks, hebasto!
In the process, we've upstreamed dozens of bugfixes and modernizations for our various build-time and run-time dependencies. Thanks especially to Michael Ford (fanquake) for that effort. In the true open-source spirit, other users of those dependencies have received some free improvements from this work.
Now that it's gone through a long round of review by the buildsystem devs, we believe it's in good enough shape to merge into master in the upcoming weeks as planned, soon after feature-freeze for v28 and branch-off for v29.
We recognize that this will be a bumpy time for Bitcoin Core dev. It's simply impossible to avoid problems with a change like this. We believe that common workflows should be mostly uninterrupted, but those of you targeting esoteric platforms or build configs are more likely to run into some issues. So this would be a good time (before merge!) for anyone who builds Bitcoin Core to give the CMake branch a try and let us know about any trouble you've run into.
Please re-read that last paragraph. This _will_ be a painful process. Please try to be constructive/productive with your bug reports.
The current PR can be found here: https://github.com/bitcoin/bitcoin/pull/30454
The build-related documentation has been updated there. Please consider that documentation to be canonical, so if there's a question that's not answered (or info that's stale), please mention it in the form of a review on the pull request so that everyone can benefit.
FAQ:
Q: Noooooooooo!!!
A: That's not a question :). Maintaining Autotools is a huge burden and it's only getting worse over time as it's virtually (though not entirely) unmaintained. As the person who introduced and maintained the current buildsystem and maintained it for years (though others like fanquake and hebasto have been more active for the last few) I'm happy to see it go. She sure was a good ship.
Q: Why CMake? Why not [my favorite buildsystem]
A: Because CMake is a perfectly reasonable choice for a modern free and open-source buildsystem with substantial momentum. And also because hebasto stepped up to actually do the work.
Q: Will Autotools continue to be supported?
A: Not in master, no. We (the buildsystem devs) don't believe it's worth the pain/effort of maintaining parallel buildsystems for a project as large as Bitcoin Core. We plan to delete everything relating to Autotools within days of merging CMake. However, we have no intention of backporting CMake to previous branches, so we'll have to maintain it to a small degree there.
Q: When/how will this affect me as a user of Bitcoin Core.
A: If don't build Bitcoin Core from source, or if you don't know what that means, you shouldn't notice any change.
Q: When/how will this affect me as a developer who interacts with Bitcoin Core?
A: For builders of Core who are not Core devs, You'll probably encounter a new way of building Core when v29 is released, sometime mid next year. At that point, rather than "./configure && make", you'll be using "cmake -S . -B build && cmake --build build". With any luck, by then, most of the bugs will have been reported and fixed, and builds will just work.
Q: When/how will this affect me as a Core-dev?
A: Today! Now! Please try out #30454 and report any issues there. Please try to avoid bikeshedding as much as possible though.
A: All relevant options have been ported over. See the docs for more details, but as a hint, the ccmake tool is very useful for listing build options.
Additionally, hebasto has been keeping up with a feature-parity table here: https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e
Q: Why is X tool or Y feature off by default?
A: We've decided to stick to the CMake way of doing things as much as possible. This way, a quick web search for "how to do Z with CMake" will probably turn up helpful and correct answers. Although, this paradigm switch is where we anticipate the most friction and frustration for users during the transition. While Autotools convention is to automatically find and enable features when possible (like enabling a feature if a library happens to exist on your system), the common CMake way is to require explicit opt-ins. This leads to fewer surprises as there's less magic happening out of sight, but may require a small upfront time investment to get back to the builds you're used to.
Q: Are all of CMake's cool features supported?
A: Some, not all. Focus has been on getting the initial port done correctly, with less effort going to fringe features and non-makefile generators. Generally speaking, single-config Makefiles have received the most attention. Ninja should just work. Multi-config builds may work, but we don't guarantee/support that just yet. MSVC should be in good shape as that's hebasto's platform of choice. XCode is known to work, but it's currently not very usable as it's unorganized and the build is full of irrelevant warnings. We'll clean those up in due time, but they're not high priority as we consider them new features as opposed to part of the initial port.
Reply all
Reply to author
Forward
0 new messages