Thanks for taking a look at this.

I'm not sure what you mean by "this patch is disabling the logic that extends edge pixels", as that is kinda the point of the patch. It is supposed to allow painting beyond the boundaries of an island, instead of extending edge pixels.

I agree that a more correct fix would extend the UVs per island instead of per triangle, but that increases code complexity a lot, and in my tests this fix still did always give better results than simply the edge extend.
Regarding spikes, notice that unlike the edge extend, this patch will only actually draw such spikes as far as the user is actually painting, so it should behave much better on sharp corners than the current edge extend method does.

I don't see how this can work correctly as a post process at all, as it would never match the brush strokes. In my view, painting and baking require completely different approaches.

One thing does have me worried from your image though... What is this thing?

Is that an artifact you got with the patch? How did it come to be there?

Ok, if the proposal is to disable the behavior, it could be OK, would be good to get user feedback that this is preferred though.

@Julien Kaspar (JulienKaspar) - could you test this patch?

Note that I'm not sure the improvement will be so noticeable from a user perspective since this mostly makes a difference when painting onto low poly faces.

The artifact is just another overlapping UV I was painting onto, nothing to worry about.

There's a bug report in T50831: Texture Paint 'Bleed' is angled along UV edges about those spikes. Some kind of pixel space bleeding to figure out how far to bleed would avoid them.

This patch seems like a good improvement by itself. Combining it with pixel space bleeding is a bit harder than just extending from the nearest pixel, but for best results it may be needed in the end.

@Campbell Barton (campbellbarton), indeed it would be nice to get some more feedback. I wrote this patch because of troubles I was having myself when texture painting, and I can say that I found them to be significant enough, even on higher density meshes. This change has generally improved results, though I still have some issues in corners, where the naive UV extension can cause slight overlap problems.

@Brecht Van Lommel (brecht), indeed the naive UV extension causes some trouble with spiking, and overlapping bleeds. It shouldn't be too complicated to simply ignore pixels further than the bleed distance, causing the corner spikes to become rounded corners instead. Is this what you mean by pixel space bleeding? In that case, it would indeed fix the spikes, but it would not resolve the issue of overlap in internal corners, as for that, the neighbouring connected polygons should be taken into account when computing the UV extension. This would fix all the seam/bleed artifacts completely, but I have not yet looked at how involved it would be to implement this. It would mostly depend on there being a simple and efficient way to access connected polygons in a UV island.

@Luca Rood (LucaRood), what I mean by pixel space bleeding is doing something similar to what baking does. Tag all the pixels covered by triangles and then do a dilate operation to find neighboring pixels that need to be filled too.

To resolve overlaps you'd find some metric to determine which triangle wins, the smallest distance to a triangle edge could work. Ideally even taking into account triangles from all islands, so you can set bleed to a high distance value without worrying about painting into unrelated islands.

This is just thinking of the best quality results without performance, making this efficient would take some work.

@Brecht Van Lommel (brecht), I don't think a pixel dilate operation would work well in this case. It would disregard brush strokes, and bring back the issue that this patch fixes. We don't want the bleed to just stretch the pixels at the edges outwards, instead, the bleed should reproduce as closely as possible what happens on the other side of the seam.

Regarding overlaps, indeed taking all triangles into account, painting with regard to the closest triangle (sort of voronoi), as you suggested, would be ideal. However, as you say, it would indeed be difficult to get this to be performant enough. To mitigate the issue, I was thinking that storing adjacent triangles for each triangle at initialisation of the paint session, could be an efficient way to fix the issue, at least in the case of connected polygons. This way a limit to where an edge should bleed could be calculated by taking the bisector between two adjacent seam edges. Actually just storing this bisector calculation at initialisation would suffice, removing the need to store adjacent triangles and redo this computation at each paint stroke.

I'll look into implementing and testing this, and will post back once I have some results.

In D4345#99718, @Luca Rood (LucaRood) wrote:

@Brecht Van Lommel (brecht), I don't think a pixel dilate operation would work well in this case. It would disregard brush strokes, and bring back the issue that this patch fixes. We don't want the bleed to just stretch the pixels at the edges outwards, instead, the bleed should reproduce as closely as possible what happens on the other side of the seam.

The dilate (maybe better term is flood fill) would be to find the pixels. The color to fill them would still be computed as in this patch.

Anyway, something based on mesh topology could be a big improvement even if it doesn't handle overlap due to other reasons.

Oh, I see what you mean. That makes sense.

I'll look into the topology based fix for now, as it will be much simpler. Something more robust (handling all overlaps), can be looked into later.

I have submitted my further changes as separate patches (D4436 and D4437), as they are really independent changes, that can selectively be applied, and are not directly related to the change made in this patch.