Page MenuHome

Sign executable and thumbnail DLL.
AbandonedPublic

Authored by Sergey Sharybin (sergey) on Oct 10 2019, 12:32 PM.

Details

Summary

This patch adds codesigning capability to the
buildbot system.

As a start the system is set up to sign binaries
on the Windows platform for the zip distribution
of a build.

On the build worker the code signing is started
prior to the actual packaging of the Blender
distribution as zip archive.

The capability relies on access to a disc location
that is accessible by both the worker and the
signing machine. The latter will be running the
script that is responsible for the signing. In the
case of a Windows platform that is winsigner.py.

For both the worker and the signer a copy of the
buildbot_codesign_config_template.py should be made
as buildbot_codesign_config.py. The contained settings
should be updated as necessary.

Both the worker and signer try to keep their work
traces cleaned up.

The signer script is supposed to run indefinitely. The
worker side is governed by a configurable timeout.

Diff Detail

Repository
rB Blender
Branch
jesterKing/codesign (branched from master)
Build Status
Buildable 5337
Build 5337: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
build_files/buildbot/slave_pack.py
114

signed_file is a bad name choice. It is a signature that the signed file exists, and not a file itself.

124

Comment is a sentence, starting with a capital letter and ending with a full-stop.

139

There is a timeout argument to build steps. For the compilation step we are using timeout of 3600 seconds, for other steps default timeout of 1200 seconds is used.

But even then it's better to have explicit check here, to help troubleshooting what exactly is going wrong.
For the same reason it is REALLY a good idea to log every step.

build_files/buildbot/winsigner.py
8

Missing \ around D:\Dev ?

8–12

Move to a configuration file.

19

Same comment as buildbot's side.

21
incantation = b"signtool sign /v /f {PFX} /t {TIMESTAMP_URL}"
23

Overdocumentation.

36

all_files ?

41
list(...) + list(...) + list(...)
43

Don't do it. Store command as list to begin with.

45

Name allfn doesn't make sense here. It points to a single file, not all of them.

47

No newline at end of file

47–48

Follow pep8.

57

allfn is bad name.

63

Same as above.

67–68

Use touch logic, which is likely yo be more atomic.

build_files/buildbot/slave_pack.py
127–128

Depends on how we want to go about it.

Bare minimum: just the blender.exe , and installer .msi this will stop the ZOMG SCARY CODE! dialogs that windows tends to pop up.
Lets be nice: Add the thumbnailer, it'll be loaded into a different process (explorer.exe) lets be nice and leave a calling card.
Allow users to verify nobody messed with the code: Sign all dll's, exe's and .pyd's that do not have a signature of their own (ie not the ms crt dlls)

Sign executable and thumbnail DLL.

This process is injected in the pack procedure
for Windows.

Waits until signed files are available.

  • pack blender.exe and BlendThumb.dll in tosign.zip and write to_sign on shared drive
  • wait for signed file to appear in designated directory.
  • unzip to original location

On the signing machine winsigner.py needs to be
running and have access to the signing certificate.

Differential Revision: https://developer.blender.org/D6036

Clean up code

  • address variable naming
  • make functions out of all possible parts
  • use pathlib throughout
  • split off configuration
Nathan Letwory (jesterking) marked 21 inline comments as done.Oct 11 2019, 12:44 PM

I think I addressed most concerns now.

Still need to ensure all necessary files are included in the signing process, but wanted to update the current patch with the latest state for further review.

Apparently updating the diff by amending makes for rather confusing inline commenting, so I marked most as done

Fix copy/paste error in variable name.

Sergey Sharybin (sergey) requested changes to this revision.Oct 11 2019, 2:29 PM

Please pay more attention to naming and anti-patterns which are being pointed out but still spreading over the code.

build_files/buildbot/buildbot_codesign.py
52

Why is this hardcoded?

55

You can find more clear and meaningful name.

71

You can also use type annotations.

83–84

pathfiles_absolute and pathfiles_relative ?

Even the latter one is not really true, since it's a list of pairs. But clearer name does exist (assuming this is really needed to store pairs, the purpose of this is unclear at this moment).

86–87
allfiles.extend(pathfiles)
117–119

Never re-purpose variables. Especially if they do change type.

filetuple.endswith

Makes no sense, tuple does not have endswith().

141

Double space.

build_files/buildbot/buildbot_config.py
1

buildbot_codesign_config.py ?

Is also a good practice to have buildbot_codesign_config_template.py in the repo which then expected to be copied locally and adjusted to a specific setup.

4

Don't think this is enough for macOS signing process. That could easily take like 30min.

Brecht will know better though.

Also, PEP8 again, spaces around operator.

This revision now requires changes to proceed.Oct 11 2019, 2:29 PM

Please also add an option to disable code signing with an argument (see my comment above).

  • Clean up code
  • Fix copy/paste error in variable name.
  • Clean-up and clarify code.
Sergey Sharybin (sergey) requested changes to this revision.Oct 28 2019, 10:06 AM
  • Option to disable code signing.
  • Use configuration template in the repository rather than a final configuration.
build_files/buildbot/buildbot_codesign.py
123–127

allfiles.extend(pathfiles)

This revision now requires changes to proceed.Oct 28 2019, 10:06 AM

Address latest requests.

  • Move signing configuration to template file.
  • Ignore the actual configuration in git.
  • Move signing flag as option to configuration.
Sergey Sharybin (sergey) requested changes to this revision.Oct 29 2019, 9:51 AM

Please go over all the comments, make sure they are addressed.

Also make sure that similar anti-patterns which are outlined by the comments are addressed in code where it was not yet explicitly pointed out.

build_files/buildbot/buildbot_codesign.py
87

self.sign_command ?

116

pathfiles.extend(path_to_search.rglob(pattern)) to avoid loop to flatten list later.

Also, why to have this Path to str conversions?

121

Remove prefix rather than replacing basedir as a substring:

  • Faster
  • More reliable (for rare cases when basedir might be a substring somewhere else)
  • More readable and understandable.
123–127

Can you finally tackle this one?

I stop my round of review here,

This revision now requires changes to proceed.Oct 29 2019, 9:51 AM
Nathan Letwory (jesterking) planned changes to this revision.Oct 30 2019, 9:07 AM
Nathan Letwory (jesterking) marked 11 inline comments as done.

Upcoming update should resolve outsanding issues.

Address review comments.

  • use type annotations
  • list extension instead of appending and flattening
  • clarify tuple usage for file listings: first member is full path, second member is archive name
  • fix formatted string usage
Sergey Sharybin (sergey) requested changes to this revision.Oct 30 2019, 11:13 AM

use type annotations

Either do it full-speed or don't (some return type and argument annotations are missing).
That being said, using annotations is a great thing to do.

Code aside, is there a dry-run mode to test the script (without having to use an actual certificate) to see all the steps are working fine and to troubleshoot VM setup and what not?

build_files/buildbot/buildbot_codesign.py
35

Use namedtuple, or:

@dataclass
class FileAndArchiveName:
  filepath: Path
  archive_name: str
38–46
def remove_prefix_if_exists(s: str, prefix: str) -> str:
    if s.startswith(prefix):
        return s[len(prefix):]
    return s

def give_fullpath_and_archive_name(
           basedir: str,
           path: pathlib.PurePath) -> FileAndArchiveName:
   strpath = str(path.resolve())
   return (strpath,
           remove_prefix_if_exists(strpath))

Boring, but more readable.

And again, is there a specific reason to go from Path to str?

39
basedir: Path
84

Comment is a full sentence, starting with a capital letter and ending with full-stop.

91–98

pathlib-ify.

119

If response_file a Path (as it should be) then simply do response_file.exists().

Generally, just stick to pathlib instead of os.path.

171

FYI: time.sleep(1) is not guaranteed to sleep for more than 1 second (in fact, it's almost never sleeps exactly 1 sec).

Better to just store a starting time and recompute time passed on every iteration.
Use monotonic clock (time.monotonic()).

202

Early output, and unindent the entire function.

209

First of all, what is "140.all" ? dll?

Second, this is too fragile. We might update to a newer compiler which will bump ABI number in the library name.
Even trickier, we might have a transitional period with two different compilers.

254

self.response_file.unlink(). Similar with zip_with_signed_files.

266

Just else. Then it will deal with links (junctions on Windows) and sockets and yadi yadi yadi.

build_files/buildbot/buildbot_codesign_config_template.py
13 ↗(On Diff #19262)
Path("...")
build_files/buildbot/slave_pack.py
109–113

Why not to pass as a configuration object?

121–122

pathlib-ify.

125

List of patterns seems to be duplicated. Move to configuration?

This revision now requires changes to proceed.Oct 30 2019, 11:13 AM

I'm fine with making all these changes, but I'd prefer we do that once this mechanism gets actually tested and used as well. This is holding up unnecessarily progression on the topic.

There is a minimal acceptable level of code quality, which also other developers from our team agree on.
If you don't agree with this or feel uncomfortable with working on that level please assign the task to someone else (me or Sybren).

Testing with self-signed certificate. For the testing I created a separate buildbot worker.

Setup:

  • create a test certificate (I used these instructions: https://stackoverflow.com/a/201277)
  • install certificate on machine that will do the (test) signing, copy the PFX file as well
  • copy the buildbot/ folder containing the winsigner.py file to the signer machine
  • ensure a drive location is shared between a buildbot worker and the signer machine. I used mapped network drives with the same drive letter for convenience
  • ensure a build is done with the patch applied. This means that the buildbot/ folder of the blender.git repo contains the necessary changes
  • ensure the config file for codesigning is copied from the template and settings are adjusted accordingly, done on both machines

Testing

  • signer machine: python winsigner.py
  • worker: ensure virtualenv for worker is active. Change to build directory. Call slave_pack.py with the correct parameters and path.
This revision now requires review to proceed.Nov 6 2019, 9:46 AM