This idea has been floating around for some time now, since OSL technically supports generation of PTX that can be consumed by OptiX, so this patch realizes that in Cycles. There were a few challenges to the actual implementation, since the OSL integration in Cycles was rather CPU focused with use of CPU pointers and globals everywhere. As such this patch includes deeper changes to the OSL integration, particularly to how closures, textures and attributes are handled:
- OSL closures are now generated using macros and a template file (see kernel/osl/closures_template.h, similar to how other data structures in Cycles are generated), so that no code duplication is necessary for CPU/GPU. The setup code for all of them was moved to standalone functions (see kernel/osl/closures_setup.h) and the conversion/flattening of the OSL closure tree to Cycles closures is now handled in a new loop using a stack instead of the previous recursive function using virtual function calls (see kernel/osl/osl.h).
- Texture handles for the GPU are just SVM texture IDs, on the CPU the existing pointers are used.
- For attributes I replaced the additional OSL attribute map and instead modified the SVM attribute map slightly so that it can be used with OSL too (instead of using a separately generated ID for lookup, the attribute name hash is used). This makes it easy to use on the GPU (since OSL already handles strings as hashes there) and also reduces memory usage.
The render services and shader manager were changed to no longer be shared across multiple Cycles instances. Unfortunately this is necessary, since the render services specify whether OSL should generate PTX or CPU code, so they have to be specific to a device (in case one session uses OSL on a CPU device and another session uses it with OptiX).- A few OSL data structure declarations are duplicated in Cycles code (see kernel/osl/types.h), so that kernels can be built without needing the OSL headers.
Strings are a bit peculiar. As mentioned, in the PTX OSL generates, they are just hashes, but as of v1.11.17.0 those hashes are referenced using global variables that need to be linked in, rather than just being passed around as hash values (this is improved in v1.12.*-dev, but for now are still stuck with v1.11.17.0), so have to magic some PTX together on the fly for this purpose (see OSLShaderManager::load_kernels in scene/osl.cpp).- The implementation currently requires OptiX rather than CUDA on the GPU, but shading is normally done in CUDA. To solve this, a new kernel_osl.ptx file was added that contains all shading kernels for OptiX (and everything else needed for OSL). This also has the advantage that when no OSL is used, all the extra stuff needed for OSL does not need to be compiled (so no change to load times), and similarily when OSL is used, no need to compile SVM kernel code.
From a user perspective, all of this should be fully transparent. So one can just load a scene using OSL shaders or add OSL scripts into a material graph, enable the "Open Shading Language" checkbox that is now visible when "GPU Compute" is used with OptiX too, and things just work.
There are just a few caveats: Not every OSL built-in intrinsic is implemented for the GPU (yet, see services.cu, the full list can be found at https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/v1.11.17.0/src/liboslexec/builtindecl.h), textures are limited since OIIO cannot be used on the GPU, so they are piped through the SVM system instead when on the GPU (no tiles or fancy interpolation/blurring/...) and tracing rays from OSL is not implemented for OptiX.
I've tried to keep the implementation agnostic from OptiX where possible, so that in theory support for the other backends can be added in the future when/if OSL adds support for them with relative ease. OSL also made several improvements to its OptiX support since the v1.11.17 release that is currently in use in Blender. But unfortunately there has been no release version of that yet, so stayed with v1.11.17 for now and just grabbed a few selective fixes and added them to osl.diff (namely https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/commit/b8ee545347b357ff5228ffe529996b06831fcc6a, https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/commit/22105d298202b6908445d7d6c5a59e43b9936128 and changes to make it find Python). There is also the problem that building OSL needs the CUDA and OptiX SDKs for this, which I wasn't quite sure how to best integrate in the dependency build environment, so for now that requires a CUDA_TOOLKIT_ROOT_DIR and OPTIX_ROOT_DIR environment variable to be set. It might make sense to make this optional somehow and only enable OSL OptiX support in Cycles when an OSL build with OptiX support is detected.
Screenshots of OSL and OptiX in action (first showing a simple OSL script node and second that textures are working):