Page MenuHome

VSE: Fix proxy images have incorrect size
ClosedPublic

Authored by Richard Antalik (ISS) on Jan 23 2021, 6:05 AM.

Details

Summary

This bug happens when media size doesn't match scene size and proxies
are enabled.

Issue is resolved mainly by passing is_proxy_image to
input_preprocess() function and calculating scale correction for
transorm and scale.

Problem was introduced in 571362642201. Previously all images were
scaled to fit into preview, so no special scaling for proxies was
necessary.

text strips scaling was also incorrect - set is_proxy_image when text
is generated with reduced resolution. When proxies are used for
preview, text strips will generate reduced resolustion image, that is
effectively a proxy. With current design we can transform this image
further, but information that this image is proxy wasn't set.

Correction factors are explained by comment, but in case of transform
feature functionality may be clearer if this functionality is split
in a way that transformation matrix is calculated first for preview size
scale and this matrix is passed to sequencer_image_transform_do_thread()
function.

Diff Detail

Repository
rB Blender
Branch
fix-proxy-preprocess (branched from master)
Build Status
Buildable 12336
Build 12336: arc lint + arc unit

Event Timeline

Richard Antalik (ISS) requested review of this revision.Jan 23 2021, 6:05 AM
Richard Antalik (ISS) created this revision.

Regarding splitting function I talked about in description, image_scale_factor is used as correction for translation, but I think that translate x/y could be represented as vector that can be just multiplied by provided scale factor matrix. question is how much readable would that function be then.

const float translate_x = transform->xofs * data->image_scale_factor + scale_to_fit_offs_x;
const float translate_y = transform->yofs * data->image_scale_factor + scale_to_fit_offs_y;

Also there is scale_to_fit_offs_x which is not scale to fit offset actually :( It works as it should, the name is not accurate though.

  • Add preview_scale_factor to ImageTransformThreadData and rename image_scale_factor

Updated patch after I have found another issue. These issues are quite time consuming to test manually and hard to catch. It would be best to update current transform tests with new transform features and use proxies as well, but proxies can't be used for rendering.

After editing a video I have found another issue with text strips. It fits into this patch, so I will update it with another change.

  • Fix text strips scaling as well - set is_proxy_image when text is generated with reduced resolution.
Richard Antalik (ISS) edited the summary of this revision. (Show Details)Jan 25 2021, 4:04 AM
source/blender/sequencer/intern/render.c
664–669

The explanation of fields should be in the structure definition, not in the assignment of the fields.

Imagine when you are dropping by to the area to investigate some issue or do some improvement and want to now what ImageTransformThreadInitData::preview_scale_factor is about. It will be super easy to see the comment if its written next to the property, and it will be rather tricky to find this explanation.

1741–1743

This is a bit unclear to me. Why do we consider such effects a proxy? Why don't we consider scene a proxy? What if it is an effect on a scene strip?

Richard Antalik (ISS) marked 2 inline comments as done.
  • Add function to check whether input image needs to be scaled to preview resolution
source/blender/sequencer/intern/render.c
601

Maybe call it seq_need_scale_to_render_size() ?

606–609

This part is still not really clear to me. Can you explain why these exceptions are needed? How about movie clips here?

source/blender/sequencer/intern/render.c
1741–1743

What I wrote in description about text effect seems to apply to all effects. Here I forgot to check for text effect only but that would be incorrect solution.

So this isn't probably good way to solve the issue.

Scenes are rendered in arbitrary resolution(set by target scene) and therefore handled as any other image.
However I should really go over all other strip types, because movieclip or mask can be affected by this.

Effects operate in preview resolution, so the input image is already in correct scale. But preprocessing assumes that any image that isn't proxy needs to be scaled to match preview scale. In other words to have same scale as proxies used.

So as a bugfix it would be probably best to move condition this into input_preprocess() and explain why effects needs special handling.

Forgot to submit comment, that should explain input_has_unity_scaling_factor()

Not sure if I am explaining it correctly. But technically this preprocessing is doing 2 scaling steps at once - scaling large images to proxy size if it is used and it applies strip transform properties.

What I wrote in description about text effect seems to apply to all effects. Here I forgot to check for text effect only but that would be incorrect solution.

It is always a good idea to put explanation to the code. Makes explanation more local (so you can clearly see where it is applied to), It also makes explanation more "permanent": after a while it will be hard to track commit message down, but comment always stays in the code.

Effects operate in preview resolution, so the input image is already in correct scale. But preprocessing assumes that any image that isn't proxy needs to be scaled to match preview scale. In other words to have same scale as proxies used.

If the effect is in preview resolution, then the scaling to preview will use scale of 1, which then can be easily early-returned, without actual computation. Sounds more generic without requiring all those tricky exception cases. At least from the looks of it.

Not sure if I am explaining it correctly. But technically this preprocessing is doing 2 scaling steps at once - scaling large images to proxy size if it is used and it applies strip transform properties.

This is fine, but you can still disambiguate naming of the function. Scaling factor is rather ambiguous, while seq_is_rendered_in_preview_resolution() or seq_need_scale_to_render_size() has more clear semantic meaning.

Richard Antalik (ISS) marked 2 inline comments as done.
  • Rename function to seq_need_scale_to_render_size()
  • Use seq_need_scale_to_render_size() for crop as well.

What I wrote in description about text effect seems to apply to all effects. Here I forgot to check for text effect only but that would be incorrect solution.

It is always a good idea to put explanation to the code. Makes explanation more local (so you can clearly see where it is applied to), It also makes explanation more "permanent": after a while it will be hard to track commit message down, but comment always stays in the code.

Right I will try to be more verbose, just sometimes I look at code and think that it is pretty obvious.
If I am missing something or something needs clarification please remind me.

Effects operate in preview resolution, so the input image is already in correct scale. But preprocessing assumes that any image that isn't proxy needs to be scaled to match preview scale. In other words to have same scale as proxies used.

If the effect is in preview resolution, then the scaling to preview will use scale of 1, which then can be easily early-returned, without actual computation. Sounds more generic without requiring all those tricky exception cases. At least from the looks of it.

It depends what you mean by early-out. in this case seq_need_scale_to_render_size() will define if "early out" is possible.
You can't define it design-wise currently, because effects can use preprocessing with all features, such as strip transform, that will do scaling to preview size and image loc rot scale at once to improve performance.
Ideally effects would use almost equivalent function as input_preprocess but clearly designed to do post-processing. That would simplify code a bit, but I probably wouldn't want to do it as bugfix.

source/blender/sequencer/intern/render.c
606–609

Effect, mask and scene in strip input strips are rendered in preview resolution. They are already downscaled. input_preprocess() does not expect this to happen.

Movie clips are rendered with original media resolution, unless proxies are enabled for them. With proxies is_proxy_image will be set correctly to true.

Thanks for the explanation.

From the code, there is easy-to-address feedback. Is there anything else you wanted to be checked, or you think all the cases are covered now?

source/blender/sequencer/intern/render.c
606–609

Add this as a comment :)

Richard Antalik (ISS) marked an inline comment as done.
  • Add comment to explain seq_need_scale_to_render_size()
source/blender/sequencer/intern/render.c
606–609

Added with slight modification "Movie clips" -> "Other strip types"

Think from the code side it's fine now.

Not sure if there is something else you wanted to check (from some of the comment in the discussion I've got an impression you wanted to check something, but then I didn't see follow-up on that).

Do you think this is something we can cover with regression test?

This revision is now accepted and ready to land.Jan 27 2021, 2:01 PM

Think from the code side it's fine now.

Not sure if there is something else you wanted to check (from some of the comment in the discussion I've got an impression you wanted to check something, but then I didn't see follow-up on that).

Do you think this is something we can cover with regression test?

Don't think there is anything left to check.
Tests would be probably useful, but this would have to be done in special way to use context with specific render size and image would have to be dumped then outside of render procedure, because that would force non-proxy context. So C based test probably. Haven't checked limitations there so far.

This revision was automatically updated to reflect the committed changes.