Page MenuHome

Add glTF library for loading OpenXR controller model
ClosedPublic

Authored by Peter Kim (muxed-reality) on Aug 30 2021, 6:27 AM.

Details

Summary

The XR_MSFT_controller_model OpenXR extension provides a glTF
controller model that can be displayed to users during a VR session. If
we can support this, it would be a big advantage for VR immersion.

Since Blender does not currently have any glTF utilities in C/C++ (only
Python), a simple and sufficient solution for loading this glTF
controller model, which will be performed at the GHOST abstraction
layer (see D10948: XR Controller Support Step 4: Controller Drawing), is needed.

The following two libraries, TinyGLTF and fx-gltf are the primary candidates:

TinyGLTF
TinyGLTF (https://github.com/syoyo/tinygltf) is a lightweight glTF
2.0 loader written in C++. It is licensed under the MIT License and
its two bundled dependencies, json (https://github.com/nlohmann/json,
MIT License) and stb (https://github.com/nothings/stb,
MIT/public domain) are similarly compliant with Blender's licensing
requirements.

fx-gltf
fx-gltf (https://github.com/jessey-git/fx-gltf) is another lightweight
glTF 2.0 loader written in C++. However, compared to TinyGLTF, it has
fewer dependencies (only json) and notable performance gains over
TinyGLTF. It is also licensed under the MIT License.

Diff Detail

Repository
rB Blender
Branch
gltf-extern (branched from master)
Build Status
Buildable 17216
Build 17216: arc lint + arc unit

Event Timeline

Peter Kim (muxed-reality) requested review of this revision.Aug 30 2021, 6:27 AM
Peter Kim (muxed-reality) created this revision.

Just some food for thought. In full disclosure I'm the author of another gltf library that could be of interest. It has fewer dependencies (no stb), is more than 3x smaller, uses the same MIT license, also makes use of nlohmann's json library, and should be similarly capable outside of Draco support[1] There are 2 downsides though: It uses exceptions (but so does GHOST) and my recent master version makes use of std::filesystem whereas Blender has just recently chosen not to do that (it was decided to use a not-quite-standard filesystem since Mac is lagging behind). It is completely up to you if you would like to try it out.

Beyond that, I wonder if moving the json library to its own extern path would be better. It would allow other parts of Blender to use json if that might happen in the future? Others would have to comment on that; decision can probably be delayed until then though.

[1] https://github.com/jessey-git/fx-gltf

Add fx-gltf, separate json lib from glTF libs

Peter Kim (muxed-reality) retitled this revision from Add TinyGLTF library for loading OpenXR controller model to Add glTF library for loading OpenXR controller model.Sep 7 2021, 11:52 AM
Peter Kim (muxed-reality) edited the summary of this revision. (Show Details)

Just some food for thought. In full disclosure I'm the author of another gltf library that could be of interest. It has fewer dependencies (no stb), is more than 3x smaller, uses the same MIT license, also makes use of nlohmann's json library, and should be similarly capable outside of Draco support[1] There are 2 downsides though: It uses exceptions (but so does GHOST) and my recent master version makes use of std::filesystem whereas Blender has just recently chosen not to do that (it was decided to use a not-quite-standard filesystem since Mac is lagging behind). It is completely up to you if you would like to try it out.

Hey, thanks for the information.
I tested loading the OpenXR controller model using fx-gltf and everything worked with no issues.
I'd be completely fine with using fx-gltf over TinyGLTF, especially with the better performance and less dependencies.
My one concern would be the use of std::filesystem, although maybe it's fine since this would be in extern - I guess the platform module members can make a decision on that.

Beyond that, I wonder if moving the json library to its own extern path would be better. It would allow other parts of Blender to use json if that might happen in the future? Others would have to comment on that; decision can probably be delayed until then though.

Thanks, I also think that's a good idea. I added the json dependency as a separate external lib.

Have you looked into using the importer we already have in python, you mention it's "not in C/C++" but not why that would disqualify it, I'd prefer not to take on any extra dependencies for functionality we already have elsewhere

Have you looked into using the importer we already have in python, you mention it's "not in C/C++" but not why that would disqualify it, I'd prefer not to take on any extra dependencies for functionality we already have elsewhere

Thanks for the feedback. Sorry, I should have been more specific about the usage.

The reason a glTF importer in C/C++ would be desirable for this use case is because the data for the OpenXR controller model is obtained as a binary buffer in memory (rather than a filepath to a GLB) via xrLoadControllerModelMSFT.
While this buffer could be wrapped as an RNA pointer, passed to the Python importer, and then have the extracted information sent back to GHOST, I think it makes more sense to keep any references to the glTF controller data contained to the single GHOST class that uses it.

In addition, there are several optimizations that are possible by doing the glTF loading directly in GHOST, including loading the data asynchronously (which may be complicated if using the Python importer) and obtaining extension properties (XrControllerModelPropertiesMSFT) for controller animation during loading.

I'm not familiar with how glTF performs in practice, but it seems we'd do a basic import of simple geometry. So I don't think performance is much of a concern. I don't care which lib we use as long as it's lightweight and easy to keep updated. The use of std::filesystem is a deal breaker though. (Although we could restrict its usage to XR only which is disabled by default on macOS anyway -- but eeeh...).

My main points of concern are

  1. We already have GLTF IO code, I'm not sure if having 2 pieces of code with similar goals is the way to go, on top of that I'd run it at-least by @Julien DUROURE (julien) to hear his thoughts on how this can be made to work without resorting to yet another library being added.
  1. I'm not convinced this sort of mesh IO belongs inside ghost, what if a different vendor uses FBX for their extension and another OBJ, or perhaps we will allow the end user to supply their own meshes in any arbitrary format are we are going to stuff all of that into ghost? same for any texture formats they may or may not use, we'd be rebuilding a completely new and separate IO stack into ghost. I can see the drawing being done in ghost, I'm much less convinced about the IO part of it.

One option to remove std::filesystem would be to pick up the prior version 1.2 (https://github.com/jessey-git/fx-gltf/tree/v1.2.0) The only difference between that version and current master is a 1-line warning fix and the filesystem support.

But yeah, before moving forward, maybe there's a better place for parsing the file - a place where the existing python code would be useable.

Can we move this forward? The more VR stuff we get in before the bcon2 switch, the better.

  • I personally don't think it's much of an issue to have a lightweight glTF library in extern/, even if there is a bundled Add-on also implementing that functionality. Making sure we use the same code the add-on uses sounds like it could cause trouble while not solving much of an issue. But yeah, curious what @Julien DUROURE (julien) has to say.
  • @Jesse Yurkovich (deadpin), I don't think we should use outdated versions of libraries that won't get fixes/updates. Since the plan is to have this library in extern/ it's easy to change later on if wanted.
  • D12544: BlenLib: Add JSON Serialization/Deserialization Abstraction Layer. also proposes to introduce nlohmann/json.

@Peter Kim (muxed-reality) there's no need to keep the license and README.md files AFAIK. We use a tool to extract all library licenses into release/license.

Hello,
for loadinf glTF files, the glTF-Blender-IO addon uses:

  • standard python
    • json for loading json part of file
    • struct for loading binary data
  • numpy (already packaged in python blender version) for data manipulation
  • bpy for any blender stuff creation / retrieving

Not sure how using same code in addon & core can work. This probably means having some wrapper / expose more data/function to bpy. Not sure it's worth it, as @Julian Eisel (Severin) says.
I don't have any opinion about C/C++ code, I am not really aware of how this work in blender source.

Another thing regarding the glTF add-on, I guess this is focused on reading glTF files into Blender data, so objects, materials, animations and such. This is different from what we need, which is just a way to read glTF files to get basic buffers out of it.

Update based on discussion

Thanks everyone for your feedback.
I will be adding @Julian Eisel (Severin) as a reviewer, since there are currently no reviewers. Feel free to add additional reviewers as you see necessary.
Also feel free to adjust the project tags if you feel like some projects should or shouldn't be tagged.

I'm going to give Ray the chance to intervene once more. Personally I think this is fine.

This revision is now accepted and ready to land.Sep 21 2021, 6:04 PM

I'm here to guide, not stand in the way, if you are both happy with this direction, make it so!

For TinyGLTF I think you can #define TINYGLTF_NO_STB_IMAGE and TINYGLTF_NO_STB_IMAGE_WRITE so you don't have to include the stb_* image stuff. That could simply and shrink this dependency further.

Remove stb dependencies

For TinyGLTF I think you can #define TINYGLTF_NO_STB_IMAGE and TINYGLTF_NO_STB_IMAGE_WRITE so you don't have to include the stb_* image stuff. That could simply and shrink this dependency further.

Thanks! That worked just fine.
Will commit this after D12567: Blender Libraries: Add JSON Library..

Okay, this seems simple enough, so let’s get it in