Blender as a Python Module - Extension Module Packaging Script
Needs ReviewPublic
Actions

Authored by Tyler Alden Gubala (TylerGubala) on Oct 16 2020, 3:39 AM.

Details

Reviewers

Campbell Barton (campbellbarton)

Summary

Users can't simply install bpy when it comes out of the build process, they have to manually copy to the right spot.

This "solves" that with python3 setup.py install that lets the user quickly take the files with them.

The readme.rst would need to be copied into the build directory, as well as the bpy_module build files.

I don't really know how to do that in CMake and am looking for any help I can get as I am very novice.

Thank you in advance.

Diff Detail

Event Timeline

Tyler Alden Gubala (TylerGubala) requested review of this revision.Oct 16 2020, 3:39 AM

Tyler Alden Gubala (TylerGubala) created this revision.

This patch seems to revert a couple recent changes please make sure you made the diff file correctly.

Aaron Carlisle (Blendify) added a reviewer: Campbell Barton (campbellbarton).Oct 16 2020, 10:37 AM

You probably wanted to use three dots.

git diff master...HEAD

while you're at it, also add -U1000 to the command to create a diff with enough context of the code around it.

The original diff was not meant to contain a bunch of roll backs to commits made by other authors, just the four affected Python-packaging related files.

The updated diff should reflect that correctly now, with just the new directory, setup.py, __init__.py, pre_uninstall.py and post_install.py

I removed some of the cruft and fixed a bug in the last patch (I used shutil.copy instead of shutil.copytree for copying a directory and copytree is the only valid one for a directory.

Is anything further necessary on my part?

Mateusz Grzeliński (grzelins) awarded a token.Nov 12 2020, 11:14 AM

Hey, I gave it a read, had some questions. I am not that advanced blender dev so keep in mind I can also be wrong, just friendly discussion.

Also you can set your CMAKE_INSTALL_PATH to install directly to python environment, how does it relate to this patch?

build_files/utils/bpy_module/blender/__init__.py
34	IMO there is no need for new subclass. Simple is beautifull. `Exception("reason...")` carries the same amount of information without subclassing. Look at amount of exception types that Python defines, there are not a lot of them. You usually define new type when you want to catch it.
45	Are you sure you can do that? If I understand correctly you try to find `blender/release/scripts` by finding subdirs `"datafiles", "scripts"` and this way you do not need to create the whole config directory. In know the correct file structure for blender python package is: `site-packages/2.90` `site-packages/bpy.so` (linux) There is some `sys.path` and `sys.modules` trickery done for example in `bpy.__init__.py` to find all required submodules: sys.path.extend([ join(dirname(dirname(dirname(__file__))), "addons", "modules"), join(utils.user_resource('SCRIPTS'), "addons", "modules"), ]) I only gave this patch a quick read, just asking.
build_files/utils/bpy_module/setup.py
207	Do you want to hardcode it, or use `bpy.app.version`?
244	Are you sure it requires only `numpy`? I remember I had errors regarding `requests` when I was building blender.

Thanks for the interest @Mateusz Grzeliński (grzelins) . I tried to address your questions, but I'm afraid I'm not terribly familiar with this web interface so I'm not sure if I should submit a new diff file (thus overwriting the old one) or what.

Either way I will get back with a few of the changes since they are good suggestions.

build_files/utils/bpy_module/blender/__init__.py
34	Fair enough, I will remove this.
45	I am looking for the `2.90` directory (or whatever blender version). Basically we need to find that directory, which contains all of Blender's scripts, and copy it to the hardcoded path Blender expects it. For instance on MacOS, for some reason it's required to be in a folder called `Resources`. The `sys.path` and `sys.modules` trickery that Blender does is really confusing to me, but that is part of a whole other discussion.
build_files/utils/bpy_module/setup.py
207	Since this is the `setup.py` script that installs `bpy` into the current Python environment, the module is not available to us for `import` yet and therefore we would have trouble accessing the member `bpy.app.version`, even though I do want to use the actual version name. Any corrections or ideas here are welcome.
244	I don't recall having any `requests` related errors when building; could you give an example? `numpy` is the only thing required, for instance, when [installing from `wheel`](https://github.com/TylerGubala/blenderpy/releases)

Removed unnecessary Exception subclasses
minor formatting & whitespace

I will answer technical comments later, but first:

Is this script meant to be run manually from console, or as a part of build process? As far as I know, cmake can run external commands. If you run it from cmake you can probably use variables from cmake
If you change CMAKE_INSTALL_PREFIX to your python site-packages this bpy will install properly. Also if you set INSTALL_PORTABLE=OFF and build bpy, it will install to the python environment that you have set in cmake configuration. Did you explore those options? I have it set up on other computer, I will check exact variable names and configuration and edit this post if necessary

Hi,

As far as running the wheel build automatically as part of the build, that would be interesting to do in my opinion, but it is a logically separate step from building Blender as a Python module from sources. So I guess that depends on what Blender Foundation wants to support Python build wise. In my experience Blender Foundation hasn't been too enthusiastic about supporting the Python module, and that's not meant in offence to anyone I am just bringing it up for consideration.

In terms of what it would technically take to build the wheel as part of the cmake process it would involve invoking a Python venv, installing up to date pip, setuptools, and wheel. then building the wheel using pip, like so (Windows assumed):

`py -m venv <path_to_temporary_folder>
<path_to_temporary_folder>\Scripts\activate
py -m pip install -U pip setuptools wheel
py setup.py bdist_wheel

With regards to the CMAKE_INSTALL_PREFIX, I would like to point out that Python venvs on Windows are laid out in a different directory structure from system installed Python. So the simple CMAKE_INSTALL_PREFIX probably won't work for that (albeit, I have yet to check, but another issue, https://developer.blender.org/T83260, is in my way).

Besides that, when someone makes a bpy.pyd build with different capabilities, using the CMAKE_INSTALL_PREFIX will just install that bpy.pyd installation for the user / machine / environment that was specified, whereas with a setup.py file, you will be able to enter a venv like this and install it anywhere (Windows assumed):

py -m venv venv
py -m pip install -U pip setuptools wheel
py build/windows/bin/Release/setup.py

So now on that machine you can install to any number of venvs.

On the other side you can make a *.whl file and distribute to any number of users on other machines with the same architecture.

These are just considerations and feel free to override. Thanks for your interest!

hi, Thanks for the patch! I recently fixed some bpy install related issues in T86579: macOS: improve bpy build system support. Linux had the best install to correct location support among all platforms (at the time of writing this patch too) and that's what I tried to match in my commits. So which platform is this patch fixing the install issue for ? (I don't see @Ray Molenkamp (LazyDodo) in reviewers/subscribers for windows)
If anything is remaining I suggest we fix it in CMake itself as it was easy enough. I could help out in https://blender.chat/channel/blender-builds.