Page MenuHome
Differential D12777

Bump Python to 3.9.7, add zstandard package, and bump other packages to latest versions
ClosedPublic
Actions

Authored by Sybren A. Stüvel (sybren) on Oct 7 2021, 3:49 PM.

Details

Reviewers
Ray Molenkamp (LazyDodo)
Bastien Montagne (mont29)
Brecht Van Lommel (brecht)
Maniphest Tasks
T88438: Libraries Changes for Blender 3.0
Commits
rBa5917175d8c1: Deps: Python, bundle zstandard package
rB417ce7ffc1bb: Deps: Python, bump bundled packages to their latest versions
rBf81f56751aaf: Deps: Bump Python 3.9.2 → 3.9.7
Summary

These changes will be committed in separate consecutive commits.
It's easier to review & build as a whole, hence the single diff.

Bump Python 3.9.2 → 3.9.7

If we're going to add a Python package to Blender, this usually means
rebuilding all of Python, so we might just as well bump it to the
latest 3.9 release.

Bump Python packages

certifi 2020.12.5 → 2021.10.8
chardet 4.0.0 → charset-normalizer 2.0.6
cython 0.29.21 → 0.29.24
idna 2.10 → 3.2
numpy 1.19.5 → 1.21.2
requests 2.25.1 → 2.26.0
urllib3 1.26.3 → 1.26.7

Bundle the zstandard Python package

This package allows Python scripts to handle compressed blend files.
This is for example needed by Blender Asset Tracer to send files to a
Flamenco render farm.

This change includes a new WITH_PYTHON_INSTALL_ZSTANDARD build-time
option, to control whether to actually install the package. For this
the already-existing approach for Requests was copied. I'm not too
happy with the way Python packages and their dependents (Audaspace &
Mantaflow) are handled, but that's for a different cleanup commit.

Diff Detail

Repository
rB Blender
Branch
arcpatch-D12777 (branched from master)
Build Status
Buildable 17745
Build 17745: arc lint + arc unit

Event Timeline

Sybren A. Stüvel (sybren) requested review of this revision.Oct 7 2021, 3:49 PM
Sybren A. Stüvel (sybren) created this revision.
Harbormaster completed remote builds in B17628: Diff 43012.Oct 7 2021, 3:49 PM
Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)Oct 7 2021, 3:50 PM
Sybren A. Stüvel (sybren) added a project: Core.
Ray Molenkamp (LazyDodo) added a comment.Oct 7 2021, 3:55 PM

We're not bumping of the other packages? (requests and friends etc) (does it need a new cert bundle? like we did for some of our other stuff due to the letsecrypt thing expiring?)

Sergey Sharybin (sergey) added a comment.Oct 7 2021, 3:57 PM

It might worth bumping requests library version for an updated CA certificates file.

Sybren A. Stüvel (sybren) added a comment.Oct 11 2021, 10:48 AM

I installed requests in a brand new virtualenv, just to see which dependencies it would pull in. Apparently nowadays requests no longer depends on chardet but on charset-normalizer. That project describes itself as:

A library that helps you read text from an unknown charset encoding.
Motivated by chardet, I'm trying to resolve the issue by taking a new approach. All IANA character set names for which the Python core library provides codecs are supported.

To me it looks quite nice, but this also means some more changes in this patch.

Sybren A. Stüvel (sybren) updated this revision to Diff 43125.Oct 11 2021, 11:23 AM
  • Bump requests, certifi, urllib, idna, cython, and numpy packages. Replace chardet with charset-normalizer (dependency change of requests)
  • Remove numpy patch, as it no longer applies; the option it tries to disable seems to have been removed from numpy. The patch was introduced in rB9715ad5acad9 without explicit motivation.
Harbormaster completed remote builds in B17712: Diff 43125.Oct 11 2021, 11:23 AM
Sybren A. Stüvel (sybren) retitled this revision from Bump Python to 3.9.7 and add zstandard package to Bump Python to 3.9.7, add zstandard package, and bump other packages to latest versions.Oct 11 2021, 11:32 AM
Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)
Bastien Montagne (mont29) requested changes to this revision.Oct 11 2021, 3:19 PM

Some minor but important required changes for install_deps.sh part.

build_files/build_environment/install_deps.sh
394

Should be higher, probably 4.0?

1151

Charset then? Or even Charset Normalizer

This revision now requires changes to proceed.Oct 11 2021, 3:19 PM
Sybren A. Stüvel (sybren) updated this revision to Diff 43135.Oct 11 2021, 3:27 PM
Sybren A. Stüvel (sybren) marked an inline comment as done.
Harbormaster completed remote builds in B17719: Diff 43135.Oct 11 2021, 3:27 PM
Sybren A. Stüvel (sybren) marked an inline comment as done.Oct 11 2021, 3:27 PM
Bastien Montagne (mont29) accepted this revision.Oct 11 2021, 3:33 PM

install_deps parts LGTM.

This revision is now accepted and ready to land.Oct 11 2021, 3:33 PM
Sybren A. Stüvel (sybren) added 2 blocking reviewer(s): Ray Molenkamp (LazyDodo), Sebastián Barschkis (sebbas).Oct 11 2021, 4:26 PM
This revision now requires review to proceed.Oct 11 2021, 4:26 PM
Ray Molenkamp (LazyDodo) added a comment.Oct 11 2021, 11:14 PM

Having some build issues with zstandard which i'm unsure how to resolve quickly (it's trying to build release bindings for the debug configuration, which is missing the release python libs, and you get a neat linker error) however while looking into that it seems that it implicitly drags in pep517 and it (or its deps) drag in tomli, pyparsing, packaging, colorama, build which doesn't sit super well with me.

I'll be travelling till sunday so my availability over the next week is sadly going to be non-existent, I'll do my best to get this all working before bcon3 but cannot make any guarantees here

Ray Molenkamp (LazyDodo) updated this revision to Diff 43173.Oct 12 2021, 3:57 AM
  • Build fixes for windows.
Harbormaster completed remote builds in B17745: Diff 43173.Oct 12 2021, 3:57 AM
Brecht Van Lommel (brecht) added a task: T88438: Libraries Changes for Blender 3.0.Oct 12 2021, 3:39 PM
Brecht Van Lommel (brecht) added a subscriber: Brecht Van Lommel (brecht).Oct 12 2021, 3:47 PM

I guess pep517 is only a build time dependency? Still would need to check we don't ship all those dependencies.

Ray Molenkamp (LazyDodo) accepted this revision.Oct 13 2021, 4:09 AM
Ray Molenkamp (LazyDodo) mentioned this in T88438: Libraries Changes for Blender 3.0.

Validated none of the deps i mentioned are in the site-packages folder, so all good there (atleast on windows) builds and passes tests on windows.

Brecht Van Lommel (brecht) accepted this revision.Oct 18 2021, 4:22 PM
Brecht Van Lommel (brecht) removed a reviewer: Sebastián Barschkis (sebbas).

Accepting as the macOS maintainer, I'll build the libraries.

This revision is now accepted and ready to land.Oct 18 2021, 4:22 PM
Brecht Van Lommel (brecht) added a comment.Oct 19 2021, 9:32 AM

I added the following to T88438, to ensure the addition of zstandard is tested.

  • Update bundled modules test, after all platforms are updated (P2515)
Brecht Van Lommel (brecht) mentioned this in rBL62724: macOS Arm: update Python, bundled modules and add zstandard.Oct 19 2021, 10:05 AM
Brecht Van Lommel (brecht) mentioned this in rBL62725: macOS Intel: update Python, bundled modules and add zstandard.Oct 19 2021, 12:37 PM
Closed by commit rBf81f56751aaf: Deps: Bump Python 3.9.2 → 3.9.7 (authored by Sybren A. Stüvel (sybren)). · Explain WhyOct 21 2021, 1:49 PM
This revision was automatically updated to reflect the committed changes.
Sybren A. Stüvel (sybren) added a commit: rBf81f56751aaf: Deps: Bump Python 3.9.2 → 3.9.7.
Sybren A. Stüvel (sybren) added a commit: rB417ce7ffc1bb: Deps: Python, bump bundled packages to their latest versions.
Sybren A. Stüvel (sybren) added a commit: rBa5917175d8c1: Deps: Python, bundle zstandard package.
Sybren A. Stüvel (sybren) added a comment.Oct 21 2021, 2:08 PM

Note: the patch as displayed here now is not what was reviewed. After reviewing I split the patch into three commits, and Phabricator auto-updated the patch to match the first of those three.

Sybren A. Stüvel (sybren) mentioned this in rBL62730: Python upgrades for Blender 3.0.Oct 21 2021, 2:37 PM
Ray Molenkamp (LazyDodo) mentioned this in rBL62735: Windows: Python 3.9.7 .Oct 21 2021, 5:43 PM
Brecht Van Lommel (brecht) added a comment.Oct 21 2021, 8:32 PM

@Sybren A. Stüvel (sybren), import zstandard fails for me on Linux, probably you did not commit the .so file.

Traceback (most recent call last):
  File "<blender_console>", line 1, in <module>
  File "/home/brecht/dev/worktree_build/bin/3.0/python/lib/python3.9/site-packages/zstandard/__init__.py", line 39, in <module>
    from .backend_c import *  # type: ignore
ModuleNotFoundError: No module named 'zstandard.backend_c'
Brecht Van Lommel (brecht) mentioned this in rBaea2287af33c: Tests: updated Python bundled modules test.Oct 21 2021, 9:25 PM

Revision Contents
Changeset List

PathSize
11 lines
build_files/
build_environment/
cmake/
1 line
10 lines
23 lines
38 lines
patches/
windows/
3 lines
source/
creator/
18 lines

Diff 43173

View Options

CMakeLists.txt

Loading...
View Options

build_files/build_environment/cmake/numpy.cmake

Loading...
View Options

build_files/build_environment/cmake/python_site_packages.cmake

Loading...
View Options

build_files/build_environment/cmake/versions.cmake

Loading...
View Options

build_files/build_environment/install_deps.sh

Loading...
View Options

build_files/build_environment/patches/numpy.diff

Loading...
View Options

build_files/build_environment/windows/build_deps.cmd

Loading...
View Options

source/creator/CMakeLists.txt

Loading...