I had a vague idea about collision respecting object normals too. However, I think this setting should probably be on the collider object and not on cloth (or in both places), because what this option really specifies is whether the collider surface should be interpreted as the outer boundary of a solid object rather than something like a balloon filled with air. That obviously depends on the kind of object.

In D1710#38063, @Alexander Gavrilov (angavrilov) wrote:

I had a vague idea about collision respecting object normals too. However, I think this setting should probably be on the collider object and not on cloth (or in both places), because what this option really specifies is whether the collider surface should be interpreted as the outer boundary of a solid object rather than something like a balloon filled with air. That obviously depends on the kind of object.

You're right. I will try to add this option to the collider object. But as the structure "CollisionModifierData" does not store any flag, it's kind of complicated.

Can you give a more detailed explanation of what this is doing? A collision is detected between cloth and another object, the face normals are pointing some particular direction, and then what is different with this option? What is the effect of flipping the collision pair normal?

The RNA description "Directs the vertex normal to the outer surface" also sounds more like a description for developers that know the implementation, not something that users can understand.

In D1710#38114, @Brecht Van Lommel (brecht) wrote:

Can you give a more detailed explanation of what this is doing? A collision is detected between cloth and another object, the face normals are pointing some particular direction, and then what is different with this option? What is the effect of flipping the collision pair normal?

I do not understand how the collision code works in general, but I was able to observe a few things:

• The vector, "collpair->normal", is the vector that indicates the direction of movement of the collision point in cloth LoopTri.
• By reversing (negate) this vector, is as if bouncing off the vertices of the LoopTri.

So this code tests if the collision point is behind the face of the object and if its direction is contrary to the normal face (LoopTri). Then reverses this vector.
If the collision point is behind the face, but it relative speed is zero, nothing happens because the code does not add a new type of impulse.

The RNA description "Directs the vertex normal to the outer surface" also sounds more like a description for developers that know the implementation, not something that users can understand.

Well, English is not my native language, but I'll think of a better description.

In general the idea of normal-aware collision option would be to make collision forces treat certain collision meshes as a boundary of a solid object, rather than just a random surface. Currently, once a cloth vertex somehow gets to the other side of an object surface, the collision force immediately switches direction and starts pushing the vertex further inside. This makes it permanently stuck, and often continues pulling more neighboring vertices inside. If collision could be told to instead always push along the normal, it can help to recover small penetrations.

I don't know anything about how collision implementation works, so can't comment on the actual code.

Ok, so then the description would be something like "Assume the collision object is a closed mesh with normals pointing outside, to help resolving collisions more accurately".

Ideally the cloth collision would be robust to this kind of problem already and this option wouldn't be needed, but we know the implementation isn't perfect. So if this helps a lot it may be a reasonable option to add, I'm not really familiar with this code.

@Brecht Van Lommel (brecht), good to know :) and thanks for the help.
First, as suggested by @Alexander Gavrilov (angavrilov), I'm trying make the option "adjust_to_outer_surface" to be on the collider object and not on cloth. It would be something like:

if (collmd->flags & OBJECT_NORMAL_ADJUST) {

But I'm having trouble generating the RNA. I made up a thread in BlenderArtists to see if anyone can help me.
http://blenderartists.org/forum/showthread.php?389787-Why-I-m-not-able-to-add-properties-to-the-collision-modifier

For DNA changes the structs need to aligned so that there are no gaps, which can be done by adding some padding variables.
http://wiki.blender.org/index.php/Dev:2.5/Source/Architecture/DefineProperty#DNA

        float time_x, time_xnew;    /* cfra time of modifier */
+       char flag, pad[7];
        struct BVHTree *bvhtree;    /* bounding volume hierarchy for this cloth object */

@Daniel Genrich (genscher), you who made the initial commit of cloth modifier, you can take a look at this code?

Daniel has no time at the moment I guess (see https://developer.blender.org/T47069). So maybe Campbell would be the best choice together with Brecht? (this are just guesses, better clear it on irc when people are there so before 20 UTC ;)

Thank @mathieu menuet (bliblubli), I'll try the IRC at a better time then.
It seems that Lukas Toenne is also an expert in the matter of collision. I'll add it ...

@Lukas Tönne (lukastoenne), could you look at this patch?

Since no one else has picked this up I'll do some tests later this week and if it all works well commit it. I guess neither of us fully understands the implications of this change, but if it works as well as the examples show it's probably too good not to add.

Can you attach a .blend that with a cloth sim where this patch helps a lot, like the screenshots in the summary?

I tried a few different models but didn't actually get such collision issues without the patch.

Here's an example:

Thank you for reviewing the patch :)

Thanks for the file.

What's interesting in this file is that if you add a subdivision modifier on the collision object, and set it to Simple with 3 levels of subdivision, it solves the issue too. But I would expect that extra modifier to have no effect. Similarly, a lower resolution cloth has no problem.

So that makes me think there is a bug, maybe one that happens when the size of the faces in the cloth and collision object are different. I'm trying to track down the cause.

It's even more interesting that the cloth collapses even if you set velocity damping to 0.6 so it approaches quite slowly and initially seems to wrap around and settle just fine. Then suddenly a few verts in the middle of a collider face start moving toward it at great speed and penetrate.

Does this patch help against transient penetrations caused by collider interaction, i.e. when you have cloth trapped between two colliders and they are animated in a way that they clip into each other just a bit and then separate again?

In D1710#39000, @Alexander Gavrilov (angavrilov) wrote:

Does this patch help against transient penetrations caused by collider interaction, i.e. when you have cloth trapped between two colliders and they are animated in a way that they clip into each other just a bit and then separate again?

I guess it might help to an extent, but only if the cloth has not penetrated too far since this only affects nearby faces. You might still get a chain effect where it gradually gets untrapped, but I doubt this would work robustly. For properly handling this kind of thing you need methods from the "Untangling Cloth" paper.

In D1710#39001, @Brecht Van Lommel (brecht) wrote:

I guess it might help to an extent, but only if the cloth has not penetrated too far since this only affects nearby faces. You might still get a chain effect where it gradually gets untrapped, but I doubt this would work robustly. For properly handling this kind of thing you need methods from the "Untangling Cloth" paper.

Well, even that would be better than 100% chance of being permanently trapped. Since essentially this kind of option communicates whether the mesh is a thin surface that collides on both sides, or the boundary of a solid object and thus only has one side, it can probably be meaningful to any algorithm. At the worst it could double the margin of error.

You may be really on to something re some kind of collision bug related to mesh density. This is what happens with one extra level of subdivision and triangulate thrown on the collider:

181	184	186

The penetration happens suddenly within just 2-3 frames, and conspicuously right in the middle of the triangle faces. The black grid for comparison was baked without extra subdivision, and penetrates in the same general area but a few dozen frames later. Both are with velocity damping 0.5.

Edit: it even depends on triangulation - if I switch it from Fixed to Fixed Alternate, it penetrates earlier, in a different place, but still in the middle of a triangle. The code seems to use triangles for collision anyway, so I thought that using explicit triangulation may make investigation more informative, and that appears to be right.