Thanks for moving this to a dedicated patch!

Think it is useful feature to have, but I share the same question as you: how to setup blending in a way that we don't need a manual CPU-side inversion. Something I hope @Jeroen Bakker (jbakker) will help us with :)

I am not able to get the same results as in the animated gif. Might be that I am using the image editor for masking and its masking is implemented in the overlay engine. See overlay_edit_uv here it would be easier to implement as there is a uses a custom fragment shader (edit_uv_image_mask_frag.glsl). Here the blending logic should be added and is fairly straight forward as you have access to the mask. and can calculate the invert mask in the fragment as well.

The plan is eventually to migrate the image drawing of the movie clip editor to the overlay engine. But that is still an over issue T82153: Use draw manager for clip editor drawing. If that is done there is no need to have store the invert of the mask. But I don't think that we should add this as a prerequisite of this patch.

So the question still remains how can we setup a blend mode to do an inversion of the color/alpha.
Possible short term solutions:

• Add an own GPU_BLEND_INVERT_ALPHA blending mode. Not really in favor of adding a custom mode for this case. Changing blend modes requires a flush of the GPU pipeline.
• We could use GPU_BLEND_CUSTOM that requires a custom fragment shader to output the madd (multiply/add) components of the blend. Doable but makes the code less readable.
• GPU_SHADER_2D_IMAGE_SHUFFLE_COLOR add an additive part to the shader. (invert is the same as (*-1 + 1)

as the shader is only used in two places I would prefer the last option. gpu_shader_image_shuffle_color_frag.glsl shuffle uniform would also get an shuffleAdd vec4 uniform.
When setting up the shader we should set shuffle to (-1, 0, 0, 0) and set the shuffleAdd to (1.0, 1.0, 1.0, 1.0).
If I am not mistaken this would solve the issue.

Note that the shader also needs to be adjusted in draw_filled_lasso. Shader states can be shared so it needs to be set everywhere where the shader is used.

If you need any assistance I will be online until 16h CEST.

Catching up with the changes since taking time off..

Is the drawing part is aligned with what Jeroen was proposing?

but how can I update the Blender defaults in the shipped "example" scenes?

Think you'd need to edit release/scripts/startup/bl_app_templates_system/VFX/startup.blend.

I don't see any changes to the overlay engine so would expect that the image editor/compositor doesn't have this feature implemented yet.
If there needs some back and forth needed for that I am happy to help out.

I just recompiled this against master, still working.

So what do you want me to do?
Adding the same blending factor to the image editor?

I am not entirely sure how to edit the startup blend in a git-able way. Is that something we do last, just before committing?

Are you sure you only want to change startup file settings? What about other .blend files containing open mask editors, they should have the value initialized properly too. This can be done in versioning_300.c.
(And startup files can be modified programmatically in versioning_defaults.c, including the application template startup files.)

It's been a long time (sorry), but here is an update. have added defaults for the blend factor to:

• DNA_space_defaults.h
• versioning_300.c: I added it under the most recent version bump (is that alright?)
• versioning_defaults.c: added to the blo_update_defaults_screen(), are there other parts that it needs to be added to?

I chose a value of 0.7, because the effect becomes visible in the combined display right away, once a mask is added.

On top of that, I have merged the current master (which required some small changes with renaming and removing the const for the blend factor), I hope I haven't messed up the merge 😄

Some minor feedback on the code.

But the versioning would be nice to hear from @Sebastian Koenig (sebastian_k): would it make more sense in the versioning code to use the new default blending factor of 0.7, or would it make more sense to set the factor to 1, preserving the interface look exactly the same as it was when the file was created?

Talked to Sebastian. Lets use factor 0.7 for the new files and 1.0 for the old files.

The rest I can not really review as it is drawing. Would be cool if Jeroen reviews the drawing part.

Thanks for the updates! Getting close, at least from my side!

Wait. Think there is no way to properly do-version in this case without subversion bump.

So would need to set subverison to 13 (BLENDER_FILE_SUBVERSION in BKE_blender_version.h) and put code in the if (!MAIN_VERSION_ATLEAST(bmain, 302, 13)) {.