Details

Reviewers

Ray Molenkamp (LazyDodo)
Bastien Montagne (mont29)
Brecht Van Lommel (brecht)

Maniphest Tasks

T88438: Libraries Changes for Blender 3.0

Commits

rBa5917175d8c1: Deps: Python, bundle zstandard package
rB417ce7ffc1bb: Deps: Python, bump bundled packages to their latest versions
rBf81f56751aaf: Deps: Bump Python 3.9.2 → 3.9.7

Summary

These changes will be committed in separate consecutive commits.
It's easier to review & build as a whole, hence the single diff.

Bump Python 3.9.2 → 3.9.7

If we're going to add a Python package to Blender, this usually means
rebuilding all of Python, so we might just as well bump it to the
latest 3.9 release.

Bump Python packages

certifi 2020.12.5 → 2021.10.8
chardet 4.0.0 → charset-normalizer 2.0.6
cython 0.29.21 → 0.29.24
idna 2.10 → 3.2
numpy 1.19.5 → 1.21.2
requests 2.25.1 → 2.26.0
urllib3 1.26.3 → 1.26.7

Bundle the zstandard Python package

This package allows Python scripts to handle compressed blend files.
This is for example needed by Blender Asset Tracer to send files to a
Flamenco render farm.

This change includes a new WITH_PYTHON_INSTALL_ZSTANDARD build-time
option, to control whether to actually install the package. For this
the already-existing approach for Requests was copied. I'm not too
happy with the way Python packages and their dependents (Audaspace &
Mantaflow) are handled, but that's for a different cleanup commit.

Diff Detail

Repository

rB Blender

Branch

temp-python-zstandard (branched from master)

Build Status

Buildable 17628
Build 17628: arc lint + arc unit

Event Timeline

Sybren A. Stüvel (sybren) requested review of this revision.Oct 7 2021, 3:49 PM

Sybren A. Stüvel (sybren) created this revision.

Harbormaster completed remote builds in B17628: Diff 43012.Oct 7 2021, 3:49 PM

Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)Oct 7 2021, 3:50 PM

Sybren A. Stüvel (sybren) added a project: Core.

We're not bumping of the other packages? (requests and friends etc) (does it need a new cert bundle? like we did for some of our other stuff due to the letsecrypt thing expiring?)

It might worth bumping requests library version for an updated CA certificates file.

I installed requests in a brand new virtualenv, just to see which dependencies it would pull in. Apparently nowadays requests no longer depends on chardet but on charset-normalizer. That project describes itself as:

A library that helps you read text from an unknown charset encoding.
Motivated by chardet, I'm trying to resolve the issue by taking a new approach. All IANA character set names for which the Python core library provides codecs are supported.

To me it looks quite nice, but this also means some more changes in this patch.

Bump requests, certifi, urllib, idna, cython, and numpy packages. Replace chardet with charset-normalizer (dependency change of requests)
Remove numpy patch, as it no longer applies; the option it tries to disable seems to have been removed from numpy. The patch was introduced in rB9715ad5acad9 without explicit motivation.

Harbormaster completed remote builds in B17712: Diff 43125.Oct 11 2021, 11:23 AM

Sybren A. Stüvel (sybren) retitled this revision from Bump Python to 3.9.7 and add zstandard package to Bump Python to 3.9.7, add zstandard package, and bump other packages to latest versions.Oct 11 2021, 11:32 AM

Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)

Some minor but important required changes for install_deps.sh part.

build_files/build_environment/install_deps.sh
394	Should be higher, probably 4.0?
1151	`Charset` then? Or even `Charset Normalizer`

This revision now requires changes to proceed.Oct 11 2021, 3:19 PM

Feedback @Bastien Montagne (mont29)

Harbormaster completed remote builds in B17719: Diff 43135.Oct 11 2021, 3:27 PM

Sybren A. Stüvel (sybren) marked an inline comment as done.Oct 11 2021, 3:27 PM

install_deps parts LGTM.

This revision is now accepted and ready to land.Oct 11 2021, 3:33 PM

Sybren A. Stüvel (sybren) added 2 blocking reviewer(s): Ray Molenkamp (LazyDodo), Sebastián Barschkis (sebbas).Oct 11 2021, 4:26 PM

This revision now requires review to proceed.Oct 11 2021, 4:26 PM

Having some build issues with zstandard which i'm unsure how to resolve quickly (it's trying to build release bindings for the debug configuration, which is missing the release python libs, and you get a neat linker error) however while looking into that it seems that it implicitly drags in pep517 and it (or its deps) drag in tomli, pyparsing, packaging, colorama, build which doesn't sit super well with me.

I'll be travelling till sunday so my availability over the next week is sadly going to be non-existent, I'll do my best to get this all working before bcon3 but cannot make any guarantees here

Build fixes for windows.

Harbormaster completed remote builds in B17745: Diff 43173.Oct 12 2021, 3:57 AM

Brecht Van Lommel (brecht) added a task: T88438: Libraries Changes for Blender 3.0.Oct 12 2021, 3:39 PM

I guess pep517 is only a build time dependency? Still would need to check we don't ship all those dependencies.

Validated none of the deps i mentioned are in the site-packages folder, so all good there (atleast on windows) builds and passes tests on windows.

Accepting as the macOS maintainer, I'll build the libraries.

This revision is now accepted and ready to land.Oct 18 2021, 4:22 PM

I added the following to T88438, to ensure the addition of zstandard is tested.

Update bundled modules test, after all platforms are updated (P2515)

Brecht Van Lommel (brecht) mentioned this in rBL62724: macOS Arm: update Python, bundled modules and add zstandard.Oct 19 2021, 10:05 AM

Brecht Van Lommel (brecht) mentioned this in rBL62725: macOS Intel: update Python, bundled modules and add zstandard.Oct 19 2021, 12:37 PM

Closed by commit rBf81f56751aaf: Deps: Bump Python 3.9.2 → 3.9.7 (authored by Sybren A. Stüvel (sybren)). · Explain WhyOct 21 2021, 1:49 PM

This revision was automatically updated to reflect the committed changes.

Sybren A. Stüvel (sybren) added a commit: rBf81f56751aaf: Deps: Bump Python 3.9.2 → 3.9.7.

Sybren A. Stüvel (sybren) added a commit: rB417ce7ffc1bb: Deps: Python, bump bundled packages to their latest versions.

Sybren A. Stüvel (sybren) added a commit: rBa5917175d8c1: Deps: Python, bundle zstandard package.

Note: the patch as displayed here now is not what was reviewed. After reviewing I split the patch into three commits, and Phabricator auto-updated the patch to match the first of those three.

Sybren A. Stüvel (sybren) mentioned this in rBL62730: Python upgrades for Blender 3.0.Oct 21 2021, 2:37 PM

Ray Molenkamp (LazyDodo) mentioned this in rBL62735: Windows: Python 3.9.7 .Oct 21 2021, 5:43 PM

@Sybren A. Stüvel (sybren), import zstandard fails for me on Linux, probably you did not commit the .so file.

Traceback (most recent call last):
  File "<blender_console>", line 1, in <module>
  File "/home/brecht/dev/worktree_build/bin/3.0/python/lib/python3.9/site-packages/zstandard/__init__.py", line 39, in <module>
    from .backend_c import *  # type: ignore
ModuleNotFoundError: No module named 'zstandard.backend_c'

Brecht Van Lommel (brecht) mentioned this in rBaea2287af33c: Tests: updated Python bundled modules test.Oct 21 2021, 9:25 PM

Bump Python to 3.9.7, add zstandard package, and bump other packages to latest versions
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 43012

CMakeLists.txt

build_files/build_environment/cmake/python_site_packages.cmake

build_files/build_environment/cmake/versions.cmake

build_files/build_environment/install_deps.sh

source/creator/CMakeLists.txt

Bump Python to 3.9.7, add zstandard package, and bump other packages to latest versionsClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 43012

CMakeLists.txt

build_files/build_environment/cmake/python_site_packages.cmake

build_files/build_environment/cmake/versions.cmake

build_files/build_environment/install_deps.sh

source/creator/CMakeLists.txt

Bump Python to 3.9.7, add zstandard package, and bump other packages to latest versions
ClosedPublic
Actions

Revision Contents
Changeset List