Page MenuHome

USD volume export
ClosedPublic

Authored by Piotr Makal (pmakal) on Feb 24 2022, 8:38 PM.

Details

Summary

Patch for: T95407

This patch adds support for volume (OpenVDB) USD export.

  • Allows to export both static and animated volumes.
  • Supports volumes that have OpenVDB data from files or are generated in Blender with 'Mesh to Volume' modifier.
  • For volumes that have generated data in Blender it also exports corresponding .vdb files. Those files are saved in a new folder named "volumes".
  • Slightly changes the USD export UI panel. Relative Texture Paths becomes Relative Paths (and has separate UI box) as the functionality will now apply to both textures and volumes. Disabling of this option due to Materials checkbox being turned off has been removed.


Including test project setup in the following .zip file:

Zip archive contains:

  • test.blend - file for testing volume export,
  • explosion/ - folder containing four .vdb sequence files that test.blend references in one of the volume objects,
  • expectedExportResult/ - folder containing files that are expected to be created during USD export (for comparison).


Patch was tested by exchanging data:

  • Blender-to-Blender[1],
  • Blender-to-Houdini.

[1] Please bare in mind that there is currently a bug when importing volumes in Blender, causing duplicated OpenVDB grids. Fix is currently awaiting review: https://developer.blender.org/D14204

Diff Detail

Repository
rB Blender
Branch
arcpatch-D14193 (branched from master)
Build Status
Buildable 21996
Build 21996: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Piotr Makal (pmakal) requested review of this revision.Feb 24 2022, 8:38 PM

Couple of notes regarding code:

  • Since loading and saving volume functions need access to Main object I was exploring couple of different ideas. In this patch you can see that I'm passing bmain object through constructors and keeping it in related classes / structs in similar fashion to Depsgraph object. Other route I was exploring was something similar to what is done in USD importer - passing the Main object through functions, starting from USDHierarchyIterator::iterate_and_write up to USDVolumeWriter::do_write. After consideration I decided against it - it touched to many files and in my opinion wasn't consistent with the way Depsgraph object is treated. Last option I considered was using global access to Main (via Global object), but it's global variable... so I decided that I have better options in this case.
  • I'm not super confident about my usage of functions from BLI_path_util.h, BLI_string.h and BLI_fileops.h so if @Sybren A. Stüvel (sybren) can double check methods like USDVolumeWriter::write_vdb_file and USDVolumeWriter::write_file_path I will appreciate it a lot!

In addition to code, I'm also not sure if my way of saving associated .vdb files is correct. As mentioned in the description for this patch (and which should be visible from USDVolumeWriter::write_vdb_file method I decided to create folder with similar name to an exported .usd file name (just remove extension and add "_vdb" at the end). I don't know if Blender has any existing convention for such things. And this also goes for naming .vdb files which now they have names from corresponding object in Blender (line 120 in usd_writer_volume.cc)

I started the review, and the code looks very good so far!

@Piotr Makal (pmakal), as you mentioned in your comments, we should probably have a broader discussion as to the naming of the volumes folder, as the material and volumes exporter code are using different naming conventions as it stands: textures are exported to a textures directory, whereas volumes are exported to a <usd_file_name>_vdb directory. I think both conventions make sense, but perhaps we should pick one for consistency. I'll see if I can find any guidelines about best practices in this regard and maybe we can also raise the question in the pipeline-assets-io-module channel.

Just a couple of minor points. Additional review to follow.

source/blender/editors/io/io_usd.c
184

Perhaps change the heading Additional Options to something more descriptive, like Assets?

source/blender/io/usd/intern/usd_writer_volume.h
43

Here and elsewhere, I believe the Blender coding style for return arguments is to use the prefix r_ rather than out (i.e., r_file_path).

Some more comments. Further review to follow.

source/blender/io/usd/intern/usd_writer_volume.cc
133

Returning here will cause a memory leak, because vdb_directory_name, vdb_directory_path and vdb_file_name won't be freed.

For this reason, would it be better to rework the logic above to avoid calling BLI_strdup* to dynamically allocate the strings? Perhaps you could allocate local char array variables on the stack instead. For example,

char vdb_directory_name[FILE_MAXDIR];
BLI_strncpy(vdb_directory_name, usd_file_name, FILE_MAXDIR);
BLI_path_extension_replace(vdb_directory_name, FILE_MAXDIR, "_vdb");

and using a similar approach for the other string variables as well. I didn't compile or test that snippet, but you get the idea.

145

write_grid_relationship() could be a static function, as it doesn't need to access any USDVolumeWriter members. However, since this function is so simple, I wonder if it would be clearer to just call

usd_volume.CreateFieldRelationship(pxr::TfToken(grid_id), grid_path);

directly in the code.

source/blender/io/usd/intern/usd_writer_volume.h
45

What's interesting is that usd_volume could be a const reference:

const pxr::UsdVolVolume &usd_volume

even though it seems unintuitive. Likewise, in the following three functions, usd_grid can be declared const pxr::UsdVolOpenVDBAsset &usd_grid.

source/blender/io/usd/intern/usd_writer_volume.cc
133

To clarify my last comment, the issue is that MEM_freeN() won't get called on the dynamically allocated strings if BKE_volume_save() fails and there is an early return on line 133. Thus, another solution, of course, is to make sure to call MEM_freenN() on these strings before the early return. I was suggesting allocating the char arrays on the stack as this makes future code maintenance easier (e.g., if someone adds an early return in the future and forgets to free the strings). Also, stack allocation is less overhead than dynamic allocation, but this is a minor point. I don't feel strongly about which approach is used to address this.

Piotr Makal (pmakal) updated this revision to Diff 48942.EditedMar 6 2022, 6:46 PM
  • Renamed Additional Options to Assets
  • Changed naming convention for output parameters, replacing out_ with r_
  • Fixed possible memory leak in USDVolumeWriter::write_vdb_file
  • Changed naming for folders holding .vdb files, from <usd_file_name>_vdb to volumes
Piotr Makal (pmakal) added a comment.EditedMar 6 2022, 6:59 PM

@Piotr Makal (pmakal), as you mentioned in your comments, we should probably have a broader discussion as to the naming of the volumes folder, as the material and volumes exporter code are using different naming conventions as it stands: textures are exported to a textures directory, whereas volumes are exported to a <usd_file_name>_vdb directory. I think both conventions make sense, but perhaps we should pick one for consistency. I'll see if I can find any guidelines about best practices in this regard and maybe we can also raise the question in the pipeline-assets-io-module channel.

In updated code I decided to go with volumes name, since, as you mentioned, it's similar to what is already used for textures. And as you pointed out in pipeline-assets-io-module channel this can be extended in the future to give users ability to specify target folders if such feature request arise.

Piotr Makal (pmakal) marked 6 inline comments as done.Mar 6 2022, 9:10 PM
Piotr Makal (pmakal) edited the summary of this revision. (Show Details)Mar 7 2022, 8:16 AM

Thanks for making the changes, @Piotr Makal (pmakal)!

I have an additional minor change to suggest. The code looks very good, on the whole!

Something to think about as a future change: Do we want to support copying volumes on disk that are referenced by the USD to the volumes directory as well? This is what is currently done for textures. But, since volume data may be huge, we probably don't necessarily want to always copy by default. I'm not sure of the best approach, but I don't think this needs to be addressed immediately.

source/blender/io/usd/intern/usd_writer_volume.cc
89

This is a relatively minor point, but I don't think you need to construct a VtValue() to wrap the volume_extent argument. I.e., the above line could simply be:

usd_volume.GetExtentAttr().Set(volume_extent, timecode);

There are other places where this change could be made in the code below. I know that I haven't always been consistent in my own code on this point, but I think it's probably better to set the values directly with the required type where possible.

147

As noted above, no need to wrap grid_name with VtValue().

190

No need to wrap grid_asset_path with VtValue().

211

No need to construct VtValue().

  • Removed explicit calls to pxr::VtValue constructors

Something to think about as a future change: Do we want to support copying volumes on disk that are referenced by the USD to the volumes directory as well? This is what is currently done for textures. But, since volume data may be huge, we probably don't necessarily want to always copy by default. I'm not sure of the best approach, but I don't think this needs to be addressed immediately.

That is a fair point and indeed something to think about as a future improvement to the exporter. For now though I would like to keep it simple with this patch :)

Piotr Makal (pmakal) marked 4 inline comments as done.Mar 11 2022, 8:15 PM

Thanks for the latest round of changes, @Piotr Makal (pmakal)! The patch looks good to me, but I'd like to wait for any additional comments from @Sybren A. Stüvel (sybren) based on his review.

It would be nice to have some example file(s) to test the patch with, and some screenshots of the results. The latter will be necessary for the release notes anyway.

Tested by exchanging data between Blender and Houdini.

That's good, but I'd also expect exchange between Blenders to work. If the Blender USD importer doesn't support volumes yet, it's still nice to know whether the VDB files can be reimported relatively easily.

Piotr Makal (pmakal) edited the summary of this revision. (Show Details)
  • Replaced FILE_MAX with FILE_MAXFILE macro for BLI_strncpy operation in USDVolumeWriter::write_vdb_file to match the size of the allocated memory for vdb_file_name variable.
Piotr Makal (pmakal) edited the summary of this revision. (Show Details)Mar 18 2022, 8:57 PM

I want to add a note regarding unit test for the new functionality that this patch brings. @Michael Kowalski (makowalski) informed me on private that he currently works on USD IO unit test suite, thus volumes will await for his patch to become online.

Sybren A. Stüvel (sybren) requested changes to this revision.Mar 22 2022, 10:17 AM

I'm getting compiler errors:

blender/source/blender/io/common/intern/abstract_hierarchy_iterator_test.cc: In constructor ‘blender::io::TestingHierarchyIterator::TestingHierarchyIterator(Depsgraph*)’:
blender/source/blender/io/common/intern/abstract_hierarchy_iterator_test.cc:57:96: error: no matching function for call to ‘blender::io::AbstractHierarchyIterator::AbstractHierarchyIterator(Depsgraph*&)’
   57 |   explicit TestingHierarchyIterator(Depsgraph *depsgraph) : AbstractHierarchyIterator(depsgraph)
      |                                                                                                ^
In file included from blender/source/blender/io/common/intern/abstract_hierarchy_iterator_test.cc:3:
blender/source/blender/io/common/IO_abstract_hierarchy_iterator.h:214:12: note: candidate: ‘blender::io::AbstractHierarchyIterator::AbstractHierarchyIterator(Main*, Depsgraph*)’
  214 |   explicit AbstractHierarchyIterator(Main *bmain, Depsgraph *depsgraph);
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~
blender/source/blender/io/common/IO_abstract_hierarchy_iterator.h:214:12: note:   candidate expects 2 arguments, 1 provided
blender/source/blender/io/common/IO_abstract_hierarchy_iterator.h:193:7: note: candidate: ‘blender::io::AbstractHierarchyIterator::AbstractHierarchyIterator(const blender::io::AbstractHierarchyIterator&)’
  193 | class AbstractHierarchyIterator {
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~
blender/source/blender/io/common/IO_abstract_hierarchy_iterator.h:193:7: note:   no known conversion for argument 1 from ‘Depsgraph*’ to ‘const blender::io::AbstractHierarchyIterator&’
[170/1745] Building CXX object source/blender/io/usd/CMakeFiles/bf_usd.dir/intern/usd_reader_stage.cc.o
ninja: build stopped: subcommand failed.
This revision now requires changes to proceed.Mar 22 2022, 10:17 AM

I've gone through the code, and added a bunch of inline notes.

source/blender/editors/io/io_usd.c
184

"Assets" is a word that has many meanings, and the use here is different from what Blender considers to be "assets" (i.e. in the scope of the asset browser).

Now that this option is no longer specific to textures, I feel that the separation between this group (material options + relative paths) and the other groups (subset of objects; data types) becomes a bit clumsy. Furthermore, this option group is still disabled when the "Materials" data type is deselected (if this changed, it's not described in the patch description), making it more confusing.

source/blender/io/common/intern/abstract_hierarchy_iterator.cc
138–141

This may not be the best place to do this check. This function (AbstractHierarchyWriter::check_is_animated()) is for abstract Object-level data only. What's the reason this is done here, rather than in USDVolumeWriter::check_is_animated()?

source/blender/io/usd/intern/usd_writer_volume.cc
61–63

Why is it okay to ignore errors writing VDB files? Errors should be reported to the user. If this is not necessary, the reason should be documented.

94–96

This function is called "write VDB file", but 90% of its code is actually only constructing the path the VDB file is supposed to be written to. I think it's better to split it up into separate functions.

As an added bonus, that way the caller can see the difference between "the file path could not be constructed" and "the VDB could not be written".

99–101

What are the chances of the export file path being empty, at this point in the export process? I don't think it is okay to silently skip such a situation. Using BLI_assert_msg() is probably a better approach, if you want to do a check on usd_file_path at all.

111–113

Basically the same question as above: in which situation will the splitting of usd_file_path not be succesful? And is it really okay to silently ignore the fact that the volume cannot be exported in this case?

155

Having a negation in the condition and an else clause is a bit much. Swap the if and else blocks and drop the negation to simplify the code a bit.

155–163

This logic seems to figure out which path to write to when new_vdb_file_path is empty. Such logic seems out of place in a function that's basically named "write (to) this file path".

167–169

Same question as before: in which cases will this path be empty? Is that even legal at this point in the export stage?

167–186

This block seems to do something different from the rest of the function, so should be in its own function (single responsibility principle).

190

usd_grid is modified here, even though it's a const parameter. This is confusing, especially since the name of the function doesn't suggest that it's setting anything on the usd_grid; rather it suggests something is written (which in the context of an exporter means "to disk").

193

Same issue as above, this function modifies the usd_grid but declares it as const. Even though it might technically be allowed to do this, I don't think it's very clear, especially since the function has no documentation.

source/blender/io/usd/intern/usd_writer_volume.h
34

Since the volumes are actually saved as VDB files and not included in the exported USD file, this class behaves differently from other subclasses of USDAbstractWriter. Its behaviour should be described here.

43

It would help to have some documentation here that describes which file paths are actually used for writing the VDB files.

43

Having a non-const reference means that the parameter can be modified without this being clear at the call itself. Why not simply return std::pair<std::string, bool>?

43

Document the meaning of the return value.

47–49

This function may need some documentation, as it's unclear what is written here. The name suggests it writes a file path, but it might also write *to* a file path, in which case it's unclear what the difference is between write_vdb_file() and write_file_path().

source/blender/io/usd/intern/usd_writer_volume.cc
193

I'm afraid I'm responsible for this parameter being declared const. Piotr originally had this parameter non-const and I asked him to change this to const (here and in other similar cases of function declaration elsewhere in the code). It seems odd to me (and arguably incorrect) to require a non-const reference here when it isn't needed. I would opt for an additional comment describing the behavior of the function.

Piotr Makal (pmakal) updated this revision to Diff 50490.EditedApr 15 2022, 10:16 PM
Piotr Makal (pmakal) marked 16 inline comments as done.
Piotr Makal (pmakal) edited the summary of this revision. (Show Details)

Sorry guys for the late update and thank you for your patience. After @Sybren A. Stüvel (sybren) review there has been a lot of restructuring of the code, hopefully for the better. Most of the previous comments/suggestions have been applied. But to sum it up briefly:

  • Logic regarding constructing absolute .vdb paths and relative paths has been extracted to separate methods.
  • I removed write_grid_name and write_file_path methods (mostly due to restructuring mentioned in the point above).
  • Code handling .vdb files has been moved to resolve_vdb_file method.
  • Adjusted abstract_hierarchy_iterator_test.cc file to make tests compile.
source/blender/editors/io/io_usd.c
184
  • Word "asset" is used by USD API/documentation in such cases. I'm open to other suggestions though and will be happy to implement such.
  • In terms of UI grouping of the export options, I agree, it looks clumsy indeed. It is important to remember that doing overhaul of the UI/UX to match the quality of other importers/exporters is outside of the scope of this patch. With that being said, if you guys have some simple solution for this particular problem that can be implemented - I'm all ears.
  • Regarding disabling the checkbox - I added comment to the description of this patch. Disabling no longer apply to the relative paths option.
source/blender/io/usd/intern/usd_writer_volume.cc
99–101

I disagree with placing an assertion here. I don't see a reason for Blender to close in this situation. A proper export of volume data to USD will fail in this scenario which is why I added warning message in updated diff.

Piotr Makal (pmakal) updated this revision to Diff 50742.EditedApr 23 2022, 10:34 PM
Piotr Makal (pmakal) marked an inline comment as done.
Piotr Makal (pmakal) edited the summary of this revision. (Show Details)

I put some more thought into UI and updated the diff which brings following change:

  • Renamed "Assets" to "File References" and moved this option to separate UI box. As shown in the figure below:

Your changes look good to me, @Piotr Makal (pmakal), and I like your solution of having the File References option in its own box in the UI.

When uploading a patch, please follow the guidelines on the wiki. That'll prevent the "Context not available" between modified chunks of code. No need to reupload this patch that way, but when updating it, please include more of the context (-U1000).

The code doesn't build with current master (and the newly-built USD libraries on Linux):

blender/source/blender/io/usd/intern/usd_writer_volume.cc: In member function ‘std::optional<std::basic_string<char> > blender::io::usd::USDVolumeWriter::resolve_vdb_file(const Volume*, const Main*) const’:
blender/source/blender/io/usd/intern/usd_writer_volume.cc:94:74: warning: unused parameter ‘bmain’ [-Wunused-parameter]
   94 |                                                              const Main *bmain) const
      |                                                              ~~~~~~~~~~~~^~~~~
blender/source/blender/io/usd/intern/usd_writer_volume.cc: In member function ‘void blender::io::usd::USDVolumeWriter::write_extent_attr(usdBlender__pxrReserved__::UsdVolOpenVDBAsset&, const VolumeGrid*, const Volume*, blender::float3&, blender::float3&) const’:
blender/source/blender/io/usd/intern/usd_writer_volume.cc:212:12: error: ‘class usdBlender__pxrReserved__::UsdVolOpenVDBAsset’ has no member named ‘GetExtentAttr’
  212 |   usd_grid.GetExtentAttr().Set(grid_extent, timecode);
      |            ^~~~~~~~~~~~~
source/blender/io/usd/intern/usd_hierarchy_iterator.cc
66–68

Is there any way that this can return a different value than data->filename passed to pxr::UsdStage::CreateNew(data->filename) in usd_capi_export.cc, function export_startjob()?

I think the code itself is fine. It could just use a comment that either explains that this is the same value (and obtaining it this way prevents passing around yet another string), or an explanation how it could be different (and motivate why this approach is better).

source/blender/io/usd/intern/usd_writer_volume.cc
99–101

This is a nice explanation of why the filepath can be empty, thanks.

104–105

Flip these conditions and return early. That'll reduce the cognitive complexity.

Piotr Makal (pmakal) marked 4 inline comments as done.
Piotr Makal (pmakal) edited the summary of this revision. (Show Details)
  • Fixed compilation issue for USD 22.03 version related to removed extent attribute for volume fields.
  • Updated D14193TestFiles.zip (in patch description) with new test.usda file reflecting the change mentioned in previous point.
  • Improved if logic in USDVolumeWriter::resolve_vdb_file method.
  • Added comment in USDHierarchyIterator::get_export_file_path method, further explaining what it does.

Regarding last compilation fail - it turns out that since 22.03 USD version there has been a change to UsdVolFieldBase class inheritance (UsdGeomBoundable is no longer a parent). For whatever reason running make update wasn't updating my USD lib version, so I was sitting on 21.02 and couldn't see this error. Thanks for the heads up!

source/blender/io/usd/intern/usd_hierarchy_iterator.cc
66–68

This mostly avoids duplication, otherwise, as you mentioned, explicitly storing data->filename value as a member variable in USDHierarchyIterator class would be required. I don't think there is a way for the routine above to return different path, but I'm not USD expert, so @Michael Kowalski (makowalski) can hopefully confirm this.

source/blender/io/usd/intern/usd_writer_volume.cc
104–105

I used vdb_file_path.value_or("") instead of vdb_file_path.value() - simplified the code even more and end effect is pretty much the same.

Sybren A. Stüvel (sybren) accepted this revision.EditedMay 6 2022, 11:32 AM

Not using bmain is a good thing, but the warning could be removed too ;-)

blender/source/blender/io/usd/intern/usd_writer_volume.cc:91:74: warning: unused parameter ‘bmain’ [-Wunused-parameter]
   91 |                                                              const Main *bmain) const
      |                                                              ~~~~~~~~~~~~^~~~~

Apart from that, LGTM. No need to re-review after removing the unused parameter.

PS: I'll remove the parameter, that way I can just land the patch and get it over with ;-)

This revision is now accepted and ready to land.May 6 2022, 11:32 AM

ps: please before sending in a patch, run make format to ensure it's formatted correctly.

Sybren A. Stüvel (sybren) edited the summary of this revision. (Show Details)
  • Adjust #include formatting, so that it's consistent with other Blender code and make format
  • Remove unused bmain parameter
  • Remove trailing whitespace
This revision was automatically updated to reflect the committed changes.