Page MenuHome

Initial support of clang-tidy toolchain
ClosedPublic

Authored by Sergey Sharybin (sergey) on Jun 5 2020, 2:39 PM.

Details

Summary

Brief introduction

Clang Tidy is a Clang based "linter" tool which goal is to help
fixing typical programming errors.

It is run as a separate compile step of every file, which slows
compilation down but allows to fully analyze the file the same
way as compiler does and catch non-trivial bugprone cases.

About this patch

This change includes:

  • CMake option called WITH_CLANG_TIDY which enables Clang Tidy linter tool on all source in the source/ directory.
  • CMake module which is aimed to find latest available Clang Tidy.
  • Set of rules which allows to have Blender fully compiled without extra issues.

The goal of this change is to provide a base ground so that solving
all the warnings can happen later on, as a team effort.

Technical notes

Personally I was able to use Clang Tidy side-by-side with both
Clang and GCC compilers. With Blender's make developer I run
into some compiler arguments incompatibilities where G++ specific
arguments are passed to clang-tidy. I am not aware of a way to
separate those command line arguments, so either we need to have
some separate configuration for GCC + Clang Tidy so the arguments
do not conflict, or limit Clang Tidy to use with Clang compiler
only.

Currently there is no official way of getting Clang Tidy on macOS.

On Windows, as far as I remember, there is clang-tidy distributed
together with LLVM package. It might work to pass the executable
to CMake via CLANG_TIDY_EXECUTABLE CMake argument, but I did not
test this.

Plan of action

Step 1: Get this ground work refined and put to master branch.
Step 2: Have a team effort to address warnings.
Step 3: Profit.

Step 1 is quite technical and we can easily do it.
Step 2 can be done as Quality Friday team work.

About the warnings

In the configuration files the warnings are split into separate
categories. Here is brief explanation (as I am not sure how to
mic string with comments in YAML).

Readability group.

  • readability-uppercase-literal-suffix: think we follow lowercase.
  • readability-magic-numbers: sounds very useful, but isn't that smart and often is a false-positive.
  • readability-isolate-declaration: we do not follow single declaration per line.
  • convert-member-functions-to-static: sometimes class is more readable to be used as namespace.
  • implicit-bool-conversion: might consider enabling the warning, but a decision is to be agreed on.
  • avoid-const-params-in-decls: this is where some more research is needed. It is rather helpful to have const qualifier in the definition, but not sure it's nice idea to have different const qualifiers for scalars in declaration and definition.
  • readability-simplify-boolean-expr: seemed too noisy, but can be a candidate to be enabled.
  • readability-make-member-function-const: didn't check how it works with inheritance, and how annoying it could be during development.
  • readability-misleading-indentation: seems to be conflicting with our placement of else.

The rest of the warnings are suppressed for the transition period,
I think they all should be resolved from the code

Bugprone group.

  • bugprone-narrowing-conversions: is rather annoying to do indexed based loops using vector.size() as an upper boundary.
  • bugprone-unhandled-self-assignment: had some false-positives with an older version of clang-tidy. Need to be checked against the latest one.
  • bugprone-branch-clone: in some cases duplicate switch cases are less bugprone.
  • bugprone-macro-parentheses: we should probably enable it, but I had some false-positives in the past and is quite a noisy change.

Again, the rest of the warnings I consider to be addressed from code
as part of Friday cleanup.

Diff Detail

Repository
rB Blender
Branch
clang_tidy_integration (branched from master)
Build Status
Buildable 8405
Build 8405: arc lint + arc unit

Event Timeline

Sergey Sharybin (sergey) requested review of this revision.Jun 5 2020, 2:39 PM
Sergey Sharybin (sergey) created this revision.
Sergey Sharybin (sergey) edited the summary of this revision. (Show Details)Jun 5 2020, 2:40 PM
Sybren A. Stüvel (sybren) accepted this revision.EditedJun 5 2020, 2:59 PM

avoid-const-params-in-decls: this is where some more research is needed. It is rather helpful to have const qualifier in the definition, but not sure it's nice idea to have different const qualifiers for scalars in declaration and definition.

I think it's a good idea to enable this check. As far as I understand it, there are two ways to use const for parameters.

  1. Prevent modification of whatever is passed to the function as argument (somecall(thingy) will not modify thingy).
  2. Prevent modification of the local variable that receives the function argument (within the function, thingy will not get another value).

I feel that the latter is an implementation detail that helps in the readability of the function itself, but isn't necessarily of interest to any caller. Enabling the avoid-const-params-in-decls basically formalises that.

It's not so much about scalars vs. non-scalars; after all, pointers can be doubly-const (const float * const thingy) and cover both the two cases described above.

readability-simplify-boolean-expr: seemed too noisy, but can be a candidate to be enabled.

I'm a fan of this one.

readability-make-member-function-const: didn't check how it works with inheritance, and how annoying it could be during development.

Personally I like making things const where appliccable. To me it's often an annoyance when I can't do that because some other function that could be const is not.

Accepted regardless of what I wrote above, as regardless of exactly what is enabled/disabled I think this is a very good step forward.

This revision is now accepted and ready to land.Jun 5 2020, 2:59 PM

In general this seems like it would be useful.

Some open topics.

  • This patch currently warns but doesn't fix the code (it can automatically add braces for eg), is the intention to enable this at some point? Or support it as a way of automating the edits (but keep it off by default).
  • If it's important to get gcc flags working with clang-tidy, we could point clang-tidy to a wrapper script that filters the flags.
  • It'd be interesting to see if this could be a separate target from building, so it can be run without having to always/disable.
Campbell Barton (campbellbarton) added inline comments.
build_files/cmake/Modules/FindClangTidy.cmake
72

typo.

Brecht Van Lommel (brecht) requested changes to this revision.Jun 5 2020, 3:36 PM

We could ship clang-tidy with the precompiled libraries like we do clang-format. Maybe the next time we upgrade LLVM/Clang, it'll be easier then since llvm switched to using a single git repo for llvm, clang, clang-tools-extra.

Personally I was able to use Clang Tidy side-by-side with both Clang and GCC compilers. With Blender's make developer I run into some compiler arguments incompatibilities where G++ specific arguments are passed to clang-tidy. I am not aware of a way to separate those command line arguments, so either we need to have some separate configuration for GCC + Clang Tidy so the arguments do not conflict, or limit Clang Tidy to use with Clang compiler only.

Which developer option, which compiler arguments? I guess we need a CMake warning about this incompatibility at least.

source/CMakeLists.txt
26

This should have either REQUIRED and then give a fatal error if not found, or print a message about clang-tidy being disabled because not found.

This revision now requires changes to proceed.Jun 5 2020, 3:36 PM

@Sybren A. Stüvel (sybren) ,

In my understanding const float* will not trigger readability-avoid-const-params-in-decls. It will be triggerred on int foo(const int bar);. Having int foo(const int bar) {} in the definition is something we do want to support (so bar is not modified by accident in the middle of the function).

If we can enable readability-avoid-const-params-in-decls in a way that in definition we can still have const int foo i have no issue enabling the warning.

The rest I also agree they are handy and useful and something what feels like we should have. Is just from own experience they are a bit noisy, so would need team effort to tackle them ;)

I also feel specifics of what warnings to enable or disable we should have in case-by-case basis, maybe in blender-coders. Feel like that'd be more efficient, especially if we have fundamentals landed.

@Campbell Barton (campbellbarton) ,

This patch currently warns but doesn't fix the code (it can automatically add braces for eg), is the intention to enable this at some point? Or support it as a way of automating the edits (but keep it off by default).

I am aware there is an automated mode for clang-tidy, but I never used it. For something simple like braces it could be handy, for the rest I feel like it is to be double-checked by a developer.

If it's important to get gcc flags working with clang-tidy, we could point clang-tidy to a wrapper script that filters the flags.

There might be a way to avoid the wrapper script, or could be a way to affects flags passed to clang-tidy using CMake. Not sure.
But indeed wrapper script can be a fallback solution.

It'd be interesting to see if this could be a separate target from building, so it can be run without having to always/disable.

I personally find is easier to have separate build folder. However, if there is a non-nasty way to enable it as a target I've got nothing against it.

Fixed typo and made clang-tidy required when WITH_CLANG_TIDY is enabled

On Windows, as far as I remember, there is clang-tidy distributed together with LLVM package. It might work to pass the executable to CMake via CLANG_TIDY_EXECUTABLE CMake argument, but I did not test this.

Clang and MSVC take rather different command line parameters, i gave it a quick whirl and clang-tidy did not seem to have a good time being fed msvc switches, the clang based build may work with it but that broke recently with some of the TBB changes jeroen did, which i still need to fix one day, but i'll be honest, the clang based build emits about 150k warns whatever clang-tidy will add to that will just get lost in the noise. MS ships with some clang tidy support in the IDE but that also threw a fit on some of the options we are using.

so yeah clang-tidy on windows is currently "not great" I'd just hide the WITH_CLANG_TIDY behind a if(NOT WIN32) for now and focus on getting a ruleset that we like.

@Brecht Van Lommel (brecht), about the command line arguments.

In my case they are:

  • -Werror=format=
  • -Werror=stringop-truncation
  • -Werror=unused-but-set-variable
  • -Wformat-signedness
  • -Wimplicit-fallthrough=5
  • -Wlogical-op
  • -Wrestrict

Surely, they might be different from what make developer uses, but it
will be super annoying to only support GCC for one specific GCC configuration.

Sure, having GCC + Clang-Tidy for make developer-alike configuration
without support of adding extra CFLAGS is better than nothing, but I would
like to see more generic solution.

Only use WITH_CLANG_TIDY on Linux

maxOS is known to not have clang-tidy, at least not currently.

Windows (as Ray stated) is not there yet either.

Lets have baby steps, and gradually build support on all platforms
of our interest.

We could ship clang-tidy with the precompiled libraries like we do clang-format. Maybe the next time we upgrade LLVM/Clang, it'll be easier then since llvm switched to using a single git repo for llvm, clang, clang-tools-extra.

Maybe a bit wrong place for this discussion, but let me at least briefly dump my thoughts here, and then we can move discussion somewhere else.

I think we should have such developers tool in SVN (including clang-tidy, clang-format) so then the entire team is synchronized at versions. Is not so important for clang-tidy on Linux (is easy to install from repo), but clang-format can easily cause fight-over-style as there some minor behavior differences in corner cases.

The only thing I wouldn't mind if that is somehow decoupled from LLVM version we use for OSL. From the past experience OSL was quite conservative in LLVM requirement (in a way that OSL wanted old LLVM) and I don't feel like being blocked for clang-format and clang-tidyupdate because of OSL.

OSL changed this stance and generally supports building with the latest llvm. On windows we currently use clang-format from the svn libs, clang-tidy is somehow not build in our deps builder, will have to take a peek why that is.

Having int foo(const int bar) {} in the definition is something we do want to support (so bar is not modified by accident in the middle of the function).

+1

If we can enable readability-avoid-const-params-in-decls in a way that in definition we can still have const int foo i have no issue enabling the warning.

That would mean:

// Declaration that'll trigger the warning:
void const_scalar(const float arg);

// Declaration that is fine:
void const_scalar(float arg);

// Definition that can be used with either declaration without warning:
void const_scalar(const float arg) { … }

For the config file, you can avoid all the trailing backslashes by using a > sign in the appropriate place and then indenting the string:

Checks: >
    -*,
    readability-*,
    -readability-uppercase-literal-suffix,
    -readability-magic-numbers,
    -readability-isolate-declaration,
    -readability-convert-member-functions-to-static,
    -readability-implicit-bool-conversion,
    -readability-avoid-const-params-in-decls,
    -readability-simplify-boolean-expr,
    -readability-make-member-function-const,

    -readability-misleading-indentation,

    -readability-else-after-return,
    -readability-braces-around-statements,
    …

OSL changed this stance and generally supports building with the latest llvm. On windows we currently use clang-format from the svn libs, clang-tidy is somehow not build in our deps builder, will have to take a peek why that is.

clang-tidy is part of clang-extra-tools which is in its own repo separate from clang. At least it was until recently. In the most recent version llvm, clang and clang-extra-tools are all in the same repo.

I think we should have such developers tool in SVN (including clang-tidy, clang-format) so then the entire team is synchronized at versions. Is not so important for clang-tidy on Linux (is easy to install from repo), but clang-format can easily cause fight-over-style as there some minor behavior differences in corner cases.

Agreed.

The only thing I wouldn't mind if that is somehow decoupled from LLVM version we use for OSL. From the past experience OSL was quite conservative in LLVM requirement (in a way that OSL wanted old LLVM) and I don't feel like being blocked for clang-format and clang-tidyupdate because of OSL.

I think this is something we can do if the need arises. As @Ray Molenkamp (LazyDodo) mentioned this has not been a problem for recent OSL versions.

@Sybren A. Stüvel (sybren) That's good to know! Poked you in the chat to figure out some really stupid aspects of the change though.

@Brecht Van Lommel (brecht), I wasn't aware OSL has no issues with recent LLVM. If that't the case then indeed no need to make dependencies compilation more complicated than it should be.
Can you mention which of the feedback you are still awaiting to be addressed?

@Brecht Van Lommel (brecht), about the command line arguments.

In my case they are:

  • -Werror=format=
  • ...

Surely, they might be different from what make developer uses, but it
will be super annoying to only support GCC for one specific GCC configuration.

Sure, having GCC + Clang-Tidy for make developer-alike configuration
without support of adding extra CFLAGS is better than nothing, but I would
like to see more generic solution.

Ah, so if I understand correctly this has nothing to do with make developer and the options it enables specifically. These flags being enabled in the root CMakeLists.txt regardless of any CMake options.

I also can't think of a good way to make GCC + Clang-Tidy work with a single CMake config. So restricting it to clang seems fine.

This revision is now accepted and ready to land.Jun 5 2020, 4:55 PM

Cleanup: More clear line break

No functional changes, just makes it easier to read.

Great initiative.

Am all for this kind of dev help as well.

Trying the patch here, I get a linking error, but this seems related to clang itself (using 9.0.1 here), not the tidy tool.

/usr/bin/ld: -f may not be used without -shared
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I've never built Blender with Clang before. Building current master fails with the same error as @Bastien Montagne (mont29) is getting. However, when I disable USD, I get lots of undefined reference errors from the linker (I tried with the CentOS libs from SVN and my own GCC-compiled libs, maybe I should rebuild the dependencies with Clang too?):

...
/usr/bin/ld: lib/libextern_ceres.a(schur_eliminator_2_d_d.cc.o): in function `.omp_outlined.':
schur_eliminator_2_d_d.cc:(.text+0x26): undefined reference to `__kmpc_global_thread_num'
/usr/bin/ld: schur_eliminator_2_d_d.cc:(.text+0x87): undefined reference to `__kmpc_dispatch_init_4'
/usr/bin/ld: schur_eliminator_2_d_d.cc:(.text+0xa5): undefined reference to `__kmpc_dispatch_next_4'
/usr/bin/ld: schur_eliminator_2_d_d.cc:(.text+0xe2): undefined reference to `__kmpc_dispatch_next_4'
/usr/bin/ld: lib/libextern_ceres.a(schur_eliminator_2_d_d.cc.o): in function `.omp_outlined..6':
schur_eliminator_2_d_d.cc:(.text+0x2fc): undefined reference to `__kmpc_global_thread_num'
...

Please stick to the subject of changes in the code reviews. It is not helpful to go into discussions of subject which is not caused or affected by the specific change.

This revision was automatically updated to reflect the committed changes.