Page MenuHome

CMake: add WITH_LINKER_MOLD option for GCC/Clang on unix platforms
ClosedPublic

Authored by Campbell Barton (campbellbarton) on Jan 12 2022, 3:39 AM.

Details

Summary

Can give considerably faster linking, especially on system with many cores.

The mold linker recently reached 1.0, see: https://github.com/rui314/mold

The current stable release of GCC can't use this linker via -fuse-ld=mold,
so this patch uses the "-B" argument to add a binary directory containing an alternate
"ld" command that points to "mold" (which is part of the default mold installation).

Some timing tests for linking full builds for AMD TR 3970X:

  • BFD: 20.78 seconds.
  • LLD: 12.16 seconds.
  • GOLD: 7.21 seconds.
  • MOLD: 2.53 seconds.

NOTE: added reviewers who might be interested to test, I don't think this is an especially controversial change, but you might have some input.

Diff Detail

Repository
rB Blender
Branch
TEMP-LINK-MOLD (branched from master)
Build Status
Buildable 19958
Build 19958: arc lint + arc unit

Event Timeline

Campbell Barton (campbellbarton) requested review of this revision.Jan 12 2022, 3:39 AM
Campbell Barton (campbellbarton) created this revision.
  • Correct typo, improve messages.

Generally indeed fine and interesting and straightforward.
One thing I'm missing some explanation of why we need MOLD_ROOT and a second find_path. Quick text with an example directory structure (similar to how you wrong examples of accessing components of MOLD_BIN) could clarify things a lot.

build_files/cmake/platform/platform_unix.cmake
703–714

Why this is needed? From the sounds of the description it seems that it's enough to find mold binary get its directory and pass via -B.

  • Improve comment explanations, only use MOLD with UNIX AND NOT APPLE.
This revision is now accepted and ready to land.Jan 12 2022, 10:36 AM
build_files/cmake/platform/platform_unix.cmake
703–714

This is done because mold provides it's own bin directory (/usr/lib/mold on my system), that contains /usr/lib/mold/ld which can be used to override the systems ld.
Finding this is a little awkward since there are multiple possible prefixes /usr/local, /opt/mold ... etc. So this finds the lib directory relative to bin.
An argument can be made for searching /usr, /usr/local/ ... etc, directly however as this isn't using find_library ... the defaults for this might not be as reliable (eventually we'll pass in -fuse-ld=mold ... I don't have a strong opinion about how this works as long as it's reliable).

Updated the comment to explain this better.

I had not heard about this, and confirm the speedup, this is great..

For reference, this configuration works with Clang on Linux (the WITH_LINKER options are GCC only at the moment).

CMAKE_EXE_LINKER_FLAGS:STRING=-fuse-ld=/home/brecht/tools/mold/mold
  • Move errors early (for better readability).
  • Don't cache the variables that point to mold paths.

Support building with clang on Linux

I had not heard about this, and confirm the speedup, this is great..

For reference, this configuration works with Clang on Linux (the WITH_LINKER options are GCC only at the moment).

CMAKE_EXE_LINKER_FLAGS:STRING=-fuse-ld=/home/brecht/tools/mold/mold

Good to know, I looked into clang support but -fuse-ld in the C/C++ flags was giving many unused-argument warnings, CMAKE_EXE_LINKER_FLAGS works without warnings (we could check on using this for GCC too - if there are no down-sides).

Updated with Clang support on Linux, similar logic should apply to macOS.

Good to know, I looked into clang support but -fuse-ld in the C/C++ flags was giving many unused-argument warnings, CMAKE_EXE_LINKER_FLAGS works without warnings (we could check on using this for GCC too - if there are no down-sides).

Because linker is not used during compilation (;
string(APPEND CMAKE_CXX_FLAGS " -B \"${MOLD_BIN_DIR}\"")
This would recompile the entire codebase even though ideally only link step should be affected. Even if linker flags are needed for static libraries like libbf_blenfont.a, suggest using CMAKE_STATIC_LINKER_FLAGS.

Updated with Clang support on Linux, similar logic should apply to macOS.

https://github.com/rui314/mold/issues/245
mold is pre-alpha on macOS and doesn't get past this -dynamic error.

lld + darwinnew flag would need llvm toolchain

  • which might affect compiler detection in different generators
  • which is huge

Only viable option for mac is zld to my knowledge. Have been using it for months now and cuts link time in half.
https://github.com/michaeleisel/zld

EDIT the state of mac shouldn't block this patch.. I can propose that in a separate one.

Good to know, I looked into clang support but -fuse-ld in the C/C++ flags was giving many unused-argument warnings, CMAKE_EXE_LINKER_FLAGS works without warnings (we could check on using this for GCC too - if there are no down-sides).

Because linker is not used during compilation (;

Right, it's just that it's how it's done with GCC which doesn't raise warnings, perhaps we should be using CMAKE_*_LINKER_FLAGS instead everywhere.

If anyone knows which of the (CMAKE_{EXE/SHARED/STATIC/MODULE}_LINKER_FLAGS) are important to set, it would be good to know. Setting all of them gives errors with mold but I didn't dig into this further.

https://developer.blender.org/diffusion/B/browse/master/CMakeLists.txt;19a62203089ffa3e8866a1989aefef7774744aca$1294-1295
Only EXE I think. MODULE would affect bpy.so
intermediate static archives don't use a linker as per https://cmake.org/cmake/help/latest/command/add_link_options.html so that's out
and we don't build shared libraries so that's also out.

I think CMAKE_EXE_LINKER_FLAGS would give us practically all the speedup we care about. CMAKE_SHARED_LINKER_FLAGS and CMAKE_MODULE_LINKER_FLAGS we might as well set assuming those don't cause issues, but if not no big deal.

Campbell Barton (campbellbarton) retitled this revision from CMake: add WITH_LINKER_MOLD option for unix platforms to CMake: add WITH_LINKER_MOLD option for GCC/Clang on unix platforms.Jan 13 2022, 11:37 PM
Campbell Barton (campbellbarton) edited the summary of this revision. (Show Details)