Page MenuHome

PBVH Pixel extractor.
ClosedPublic

Authored by Jeroen Bakker (jbakker) on Mar 30 2022, 2:27 PM.

Details

Summary

This patch contains an initial pixel extractor for PBVH and an initial paint brush implementation.
PBVH is an accelleration structure blender uses internally to speed up 3d painting operations.
At this moment it is extensively used by sculpt, vertex painting and weight painting.

For the 3d texturing brush we will be using the PBVH for texture painting.
Currently PBVH is organized to work on geometry (vertices, polygons and triangles).
For texture painting this should be extended it to use pixels.

Screen recording has been done on a Mac Mini with a 6 core 3.3 GHZ Intel processor.

Scope

This patch only contains an extending uv seams to fix uv seams. This is not actually we want, but was easy to add
to make the brush usable.

Pixels are places in the PBVH_Leaf nodes. We want to introduce a special node for pixels, but that will be done
in a separate patch to keep the code review small. This reduces the painting performance when using
low and medium poly assets.

In workbench textures aren't forced to be shown. For now use Material/Rendered view.

Rasterization process

The rasterization process will generate the pixel information for a leaf node. In the future those
leaf nodes will be split up into multiple leaf nodes to increase the performance when there
isn't enough geometry. For this patch this was left out of scope.

In order to do so every polygon should be uniquely assigned to a leaf node.

For each leaf node

for each polygon
  If polygon not assigned
    assign polygon to node.

Polygons are to complicated to be used directly we have to split the polygons into triangles.

For each leaf node

for each polygon
  extract triangles from polygon.

The list of triangles can be stored inside the leaf node. The list of polygons aren't needed anymore.
Each triangle has:

poly_index.
vert_indices
delta barycentric coordinate between x steps.

Each triangle is rasterized in rows. Sequential pixels (in uv space) are stored in a single structure.

image position
barycentric coordinate of the first pixel
number of pixels
triangle index inside the leaf node.

During the performed experiments we used a fairly simple rasterization process by
finding the UV bounds of an triangle and calculate the barycentric coordinates per
pixel inside the bounds. Even for complex models and huge images this process is
normally finished within 0.5 second. It could be that we want to change this algorithm
to reduce hickups when nodes are initialized during a stroke.

Diff Detail

Repository
rB Blender
Branch
temp-T96710-pbvh-pixels
Build Status
Buildable 21624
Build 21624: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
source/blender/blenkernel/BKE_pbvh.hh
22 ↗(On Diff #50327)

Does this really need a class with operator overloading?

71 ↗(On Diff #50327)

incides -> indices

101 ↗(On Diff #50327)

It's unclear why this constructor does not initialize add_barycentric_coord_y. If it's intentional, it's unclear what the meaning of such a partially initialized class is.

113–123 ↗(On Diff #50327)

I'm not sure that storing loop indices will be a good use of memory if they can be looked up somewhere else.

175 ↗(On Diff #50327)

PackedPixelRow may be a better name, I think in software terms "package" has a different meaning.

177 ↗(On Diff #50327)

This only needs 2 floats.

186 ↗(On Diff #50327)

I don't think this typedef is helpful if it's only used in one place.

188 ↗(On Diff #50327)

There should be a comment explaining what this class is. It's not obvious what the meaning of "tile" is in this context, it appears to be UDIM tiles.

205–211 ↗(On Diff #50327)

I would move BKE_image_partial_update_mark_region out of this function and into the one place that calls it, and rename this to clear_dirty. And then add a mark_dirty that handles part of this logic:

if (pixels_painted) {
  int2 start_image_coord(encoded_pixels.start_image_coordinate.x,
                         encoded_pixels.start_image_coordinate.y);
  BLI_rcti_do_minmax_v(&tile_data->dirty_region, start_image_coord);
  BLI_rcti_do_minmax_v(&tile_data->dirty_region,
                       start_image_coord + int2(encoded_pixels.num_pixels + 1, 0));
  tile_data->flags.dirty = true;
}

That way there are methods that clearly affect the data that is contained in this class.

219 ↗(On Diff #50327)

This is unused.

229 ↗(On Diff #50327)

This is not implemented.

source/blender/blenkernel/intern/pbvh_pixels.cc
24

It's not clear to me how an ::extractor sub namespace helps.

47

Abstractions like this method I also find confusing. It's not obvious what the return of a method name tag_visited is (successfully tagged?), so in order to understand the one function that uses this return value you have to jump around the code, while it would have been simpler to just inline this code.

56

Would not use doxygen unless you do it consistently.

source/blender/blenkernel/intern/pbvh_pixels_seams.cc
13

I would not use using here, makes it hard to tell what data structure comes from where.

308

Is this only extending along the x-axis (why?), or just named confusingly?

source/blender/editors/sculpt_paint/sculpt_paint_image.cc
38

This is not used.

63

This is the kind of abstraction that confuses me, the contents of this class is not what would usually be described as a pixel.

92

Would not use goto, has a specific meaning that's not this. Just next_pixel can work.

102

store -> write for consistency with read.

150

Is there a plan to abstract this kind of thing and share code with vertex colors? Or would there be separate implementations of brushes for vertex colors and image textures?

I'm not sure how much code it would reduce overall though, if any.

150

For consistency, this typename could be ImageBuffer or the above class could be ImagePixelAccessorByte4.

178

Would consistently used either "packed" or "encoded", not a combination of both.

180–183

This kind of update check could be moved outside the inner loop, into a separate function.

278

It's not clear to me this works? It's possible for all 3 vertices to fall outside the brush radius but part of the triangle to still be within the brush radius.

You'd need to compute the bounding box of the 3 vertices and then have a test for that against the brush radius?

331

Locking is currently only used for render results and viewers, so no. If a painted image needed locking it would need to be moved outside this function so multithreading keeps working.

335

It seems more natural to loop over the tiles in the node and then find the image tile from that?

If performance is a concern there could be an cached array for mapping, but I don't think that's the case, and it's already doing a loop over the tile data anyway.

377

I would not use a function call here, you're already editing the tile data directly, might as well set the node dirty flag directly. Otherwise it seems like this function call may be used to mark the entire node dirty while that would really require more steps.

This revision now requires changes to proceed.Apr 11 2022, 5:44 PM
source/blender/blenkernel/BKE_pbvh.h
61

It might be a good idea to interface with C++ a bit differently. One thing I've done is putting stuff in the #ifdef _cplusplus:

#ifdef __cplusplus

namespace blender::pbvh::NodeData {
  struct NodeData;
}
using blender::pbvh::pixels::NodeData;

extern "C" {
#else 
struct NodeData;
#endif

This of course request the usage of struct, which is default-public. But that's what public/private keywords are for.

source/blender/blenkernel/intern/pbvh_pixels.cc
37

Is it necessary to make a subclass here?

156

I think it's better to use the tessellation that's stored inside the nodes directly. You can use my patch for that, or just iterate over the triangles directly; see the doc comments for PBVHNode.prim_indices in master (which I've clarified a bit).

source/blender/blenkernel/intern/pbvh_pixels_seams.cc
51

It might be a good idea to make isect_seg_seg_v2_simple inline.

source/blender/editors/sculpt_paint/sculpt_paint_image.cc
278

Btw don't use closest_on_tri_to_point_v3 (if you were thinking about it). It's a pretty expensive, it's used by dyntopo and is usually the most expensive operation in profiling tests.

Jeroen Bakker (jbakker) marked 31 inline comments as done.
  • Renamed PBVH_UpdatePixels to PBVH_RebuildPixels.
  • Renamed BKE_pbvh.hh BKE_pbvh_pixels.hh.
  • Remove BarycentricWeights.
  • Spelling in comments.
  • Remove Triangle struct.
  • Rename PixelsPackage to PackedPixelRow.
  • Rename TIleData to UDIMTilePixels
  • Encapsulate clear_dirty/mark_dirty.
  • Remove unused code.
  • Remve extractor namespace.
  • Removed VisitedPolygons.
  • Removed doxygen comments.
  • Use primitives stored in PBVH.
  • Remove Pixel class.
  • Renamed inner type to ImageBuffer.
  • Remove image locking.
  • Change the tile painting loop to use node data first.
  • Don't call a function to update the node state.
  • Fix quad data stored multiple times.
  • Remove indices from Triangles. Index of the triangle is in sync with prim list.
  • Remove using namespace.
  • Added support for secondary brush color.
Jeroen Bakker (jbakker) planned changes to this revision.Apr 12 2022, 3:10 PM
  • better brush testing for faces.
  • improve uv seams
source/blender/blenkernel/BKE_pbvh.hh
22 ↗(On Diff #50327)

No, not anymore. It used to have different storage classes what was useful during prototyping. I have removed the other implementations but haven't converted this back. Will do now.

71 ↗(On Diff #50327)

Also removed the second line as it is more a note to myself when we want to compress even more.

87 ↗(On Diff #50327)

Yes agree, will remove the struct.

96–97 ↗(On Diff #50327)

This can only be done when all left bottom uv coordinates of the pixels inside PixelsPackage would be on the triangle itself. During initial extraction this is the case. When extending the pixels for UV island borders this isn't the case anymore. My take on the UV seams at this moment is to have two algorithms that work together.

  • Near UV islands edges the PixelsPackages would be extended in the X direction to include all pixels that have more than 50% coverage.
  • For pixels that have less than 50% coverage would be copied from the nearest pixel of the connecting UV edge, or when no edge available the nearest pixel.

The reason for this is to reduce the random access when fixing seams and improve the quality a bit. Although the improvement hasn't been tested yet.

113–123 ↗(On Diff #50327)

Removed the poly_indices and loop_indices as they are available as primitive index.

177 ↗(On Diff #50327)

This would not work with the UV island extend. See comment above. But yes the normal case would only need 2 floats.
Eventually I want to encode these PackedPixelRow where the packages could be compressed based on the actual needed memory.
I didn't want to do this in the initial patch.

source/blender/blenkernel/intern/pbvh_pixels_seams.cc
308

Extending along the y-axis is also required, just not implemented.

source/blender/editors/sculpt_paint/sculpt_paint_image.cc
38

image is used in do_paint_pixels

150

I don't see that much benefit to share this code with vertex colors.

335

Not a concern, just the default way how I looked against code.

Jeroen Bakker (jbakker) edited the summary of this revision. (Show Details)Apr 12 2022, 3:11 PM
Jeroen Bakker (jbakker) edited the summary of this revision. (Show Details)Apr 12 2022, 3:53 PM
source/blender/editors/sculpt_paint/sculpt_paint_image.cc
278

I will remove this optimization from this patch, eventually this should happen per primitive to improve performance. Doesn't have to be part of the initial brush.

Jeroen Bakker (jbakker) marked 4 inline comments as done.Apr 13 2022, 1:39 PM
Jeroen Bakker (jbakker) added inline comments.
source/blender/blenkernel/BKE_pbvh.h
61

I didn't known of other areas that use this. For this case it would only remove a few casts. The downside is that you loose the FQN in C what could lead to misunderstanding and naming collisions. Personally I would hide it in a void* to reduce confusion.

source/blender/blenkernel/intern/pbvh_pixels_seams.cc
51

I want to replace this with an area coverage check. inlining this would IMO not help that much, there are more effective ways to optimize the branch prediction.

source/blender/editors/sculpt_paint/sculpt_image.cc
39 ↗(On Diff #50327)

Have replaced it with last_paint_canvas_key

source/blender/editors/sculpt_paint/sculpt_paint_image.cc
278

After testing it seems that the performance drops back more than expected. Will add a primitive test.

Jeroen Bakker (jbakker) marked 3 inline comments as done.
  • Remove vertex based brush testing.
  • PBVH: triangle brush test.
  • Remove msg bus subscribe.
  • Rebuild pixels when image resolution/uvmap changes.
Jeroen Bakker (jbakker) planned changes to this revision.Apr 13 2022, 1:40 PM

Still need to implement some features missing in the uv seam extractor.

  • Reverted file spece_view3d.c
  • Remove unused include statement.
  • Merge branch 'master' into temp-T96710-pbvh-pixels
Brecht Van Lommel (brecht) requested changes to this revision.Apr 14 2022, 7:44 PM

Looks much better to me now.

source/blender/blenkernel/BKE_pbvh_pixels.hh
61

Is there a reason this is stored on the node, since this gets recomputed for every step anyway?

If it's to avoid the repeated memory allocation, I'd guess that has negligible performance impact.

source/blender/blenkernel/intern/paint.c
1787–1794

I believe this code will get called when the mesh data / topology changes, and right before any sculpt/paint tools are used. So I think this should correctly refresh.

If you change the image resolution, then there will be a delay on your first stroke as it rebuilds just in time. If you change the active UV layer, that operation will be slow if this rebuild is slow.

I don't know if one is clearly better than the other really. We could always update immediately by hooking into the depsgraph for every depsgraph update, like ED_render_scene_update. But then you get a delay between e.g. typing X and Y resolutions. Or we could always postpone it to the first stroke.

I guess it's fine as is now. Maybe later an idea would be to update this in a background job or something.

source/blender/editors/sculpt_paint/sculpt_paint_image.cc
38

Sorry, I meant the lock.

206

This is the wrong role, it's for perceptually even distribution in color picking widgets. I think the brush color should just be scene linear, but maybe there's places that assume it's sRGB? sRGB is what the color picking role happens to be for our OpenColorIO config.

Creating and freeing color management processors often might also be slow, not sure.

This revision now requires changes to proceed.Apr 14 2022, 7:44 PM
Jeroen Bakker (jbakker) marked 4 inline comments as done.
  • Fix painting on UDIM tiles.
  • Removed bad import.
  • Move brush_test from triangles to do_paint_pixels.
  • Removed lock attribute from ImageData struct.
  • Color management of brush colors
  • Remove experimental uv seam bleeding
Jeroen Bakker (jbakker) edited the summary of this revision. (Show Details)
  • Only store 2 components of barycentric coordinates.
NOTE: I removed the seam extractor as that one requires more work added T97352: 3D texturing: Fix seam bleeding what would be my main focus for the upcoming days.
source/blender/blenkernel/intern/paint.c
1787–1794

I added T97344 to keep track of this. Especially with the UV seam extraction I don't see a good way to extract the data on a per node.

source/blender/editors/sculpt_paint/sculpt_paint_image.cc
206

Inside the brush colors are stored in sRGB. Other areas convert it to scene linear when needed, mostly when working with float color attributes (geometry) and float textures.
Perhaps on short notice we coult convert it to scene linear via srgb_to_linearrgb_v4 and then apply OCIO processor. and add a note above it.

The brush color is converted once per step, I didn't detect any performance issue so far.

This revision is now accepted and ready to land.Apr 15 2022, 3:48 PM
This revision was automatically updated to reflect the committed changes.