Page MenuHome

CMake : Move to more "modern cmake" use.
AbandonedPublic

Authored by Ray Molenkamp (LazyDodo) on Aug 3 2021, 10:41 PM.

Details

Reviewers
None
Summary

CMake : Move to more "modern cmake" use.

This change restructures our build scripts from
something based on global include directories to
something cmake is actually good at: propagating
build requirements from one build target to another.

So What does this diff change?

  1. PUBLIC/PRIVATE/INTERFACE options for the INC, INC_SYS and LIB

variables that are in most of our CMakeLists.txt files.

Cmake's documentation will shed more light on this
if you are interested but from a 5000 ft view these
are essentially the options:

PRIVATE - The build target, need this include/library to build
INTERFACE - Clients of this target need this include/library to build
PUBLIC - Both the build target and the clients of the library need
this include/library to build

when not specified PRIVATE is assumed for includes (same as before)
while INTERFACE is assumed for LIB (same as before, and virtually
always incorrect)

  1. A new build target bf_intern_atomic this is a header only library

Rather than sprinkling ../../../intern/atomic in all the INC variables
you can now just add PRIVATE bf_intern_atomic to the LIB section in
the CMakeLists.txt where you need it and cmake will take care the
include paths are set correctly.

  1. Properly configured build targets for bf_intern_guardedmalloc and

bf_blenlib

These were just low hanging fruit, and great demonstrations on what
changes will need to be made to a library to support this.

  1. Cleanup of INC variables for bf_intern_guardedmalloc and bf_blenlib

and bf_intern_atomic.


Upsides:

Easier dependency management, no need to fudge -DWITH_INTERNATIONAL in 46!! different CMakeLists. You can control it just from the CMakeLists for bf_blentranslation and have it propagate to any target that properly declares a dependency on it. we fixed -DWITH_INTERNATIONAL since then.
Better dependency management, we have been lazy and sloppy here, more than once if someone has issues, the fix is "Try a clean build!" Why does this solve it? We didn't tell the build system about implicit dependencies, and ended up with a build in a inconsistent state, count the number of libraries we have that have glog in their includes but not in their LIB variable, it is shockingly high, same for blenlib, same for guardedalloc, same for.....
It is a relatively unintrusive change, further improvements can be made incrementally rather than having to fix everything in one go
If we ever want to move to more shared libraries for parts of blender (guardedmalloc and clog are prime candidates for such an endeavor) having the dependencies properly expressed is essential.

Downsides:

Lower developer productivity, due to dependencies correctly being expressed (and no longer INTERFACE for every single lib) it's likely going to cause more/longer rebuilds, on the other hand you save time when you have to trouble shoot an inconsistent build and you just end up doing a clean and building all from scratch.

Diff Detail

Repository
rB Blender
Branch
tmp_gtest_cleanup_1 (branched from master)
Build Status
Buildable 16225
Build 16225: arc lint + arc unit

Event Timeline

Ray Molenkamp (LazyDodo) requested review of this revision.Aug 3 2021, 10:41 PM
Ray Molenkamp (LazyDodo) created this revision.
Ray Molenkamp (LazyDodo) planned changes to this revision.Aug 3 2021, 10:42 PM
  • disable absolute path checking
Ray Molenkamp (LazyDodo) planned changes to this revision.Aug 3 2021, 11:00 PM
Ray Molenkamp (LazyDodo) retitled this revision from CMake : Fix include dirs mess to CMake : Move to more "modern cmake" use..Aug 4 2021, 1:03 AM
Ray Molenkamp (LazyDodo) edited the summary of this revision. (Show Details)
Ray Molenkamp (LazyDodo) edited the summary of this revision. (Show Details)

Some rough timing tests.

Build typebeforeafter
Full56.4068.37
Full Cached (ccache)4.907.50
Single file change (creator.c)4.404.40

Since the slow down doesn't impact rebuilding after a few files change, I don't see any problems sacrificing some performance for correctness.

The implementation looks good to me generally, have not looked at every line.

But why would this affect performance of a full build?

Ray Molenkamp (LazyDodo) edited the summary of this revision. (Show Details)
  • cleanup makesdna/rna

    A dependency on bf_blenlib got added to makesrna by accident leading to excessive rebuilds of bf_rna and bf_intern_cycles
  • Merge remote-tracking branch 'origin/master' into tmp_gtest_cleanup_1

Full I'd expect the cmake generate stage to be a little longer, as we add more dependencies and cmake has to do a little more work propagating information (Guessing this be in the ms to seconds range though currently)

as we express more of the implicit dependencies we'd be coming closer to to the "true" dependency graph (extracted from the object files, red lines are directly circular dependencies) of blender

and i honestly can't fault CMake for going "guys, i'm gonna need a few more seconds" there, as it is pretty far from an ideal architecture

Update to latest master

passes on all bots unsure who to assign for review though, @Campbell Barton (campbellbarton) the build system officially belongs to you mind taking a stab here?

I looked at the implementation more closely now, all of the changes look fine. If this gets rebased and passes tests on the buildbot, I think it's fine to commit.

Ray Molenkamp (LazyDodo) edited the summary of this revision. (Show Details)

update to latest master

given rebase is essentially "do all changes again manually"
this patch is too much to handle for me with all changes
previously included, so included are

  • the the core changes
  • the bf_intern_atomic modernisation

the blenlib and guardedalloc cleanup previously included
have only been done for the projects that really need it
(ie makesdna/rna) further clean-up can be later be done in
separate patches.

Ray Molenkamp (LazyDodo) planned changes to this revision.Jul 26 2022, 3:04 AM

https://builder.blender.org/admin/#/builders/18/builds/553

Have to figure out wtf is up with USD on windows, mac-arm timing out was happening on master as well today, so i'm counting that as a pass :)

archived to prepare for migration