Tests files are based on test from D8393
Test files should be in lib\tests\sequence_editing
These are files, I will add few more tests including animation test.
Using generic tool to compare rendered vs reference image as other render engines.
Details
Details
- Reviewers
Sergey Sharybin (sergey) - Commits
- rBc887a50f62a7: Add sequencer transform tests
Diff Detail
Diff Detail
- Repository
- rB Blender
- Branch
- transform-test (branched from master)
- Build Status
Buildable 11015 Build 11015: arc lint + arc unit
Event Timeline
Comment Actions
I am not sure if I should somehow make clear that these test files must be created with certain Blender version. I assume we would like to use these for versioning as well? But that said, I don't think this is really necessary - as long as we can confirm, that versioning code works now, we can make reference images even after versioning. That way I could include more tests like for rotation and scale.
| tests/python/CMakeLists.txt | ||
|---|---|---|
| 713 | Perhaps I don't need to do foreach with 1 test? Not sure how much tests we will have. But We should probably add tests for proxies, because those are easily breakable. | |
Comment Actions
The code seems fine to me.
For the files we kind of were tempting to use underscore instead of space, just to make automated handling easier. But is not so big of a deal.
Is there a conceptual difference between 480x270 tests and 200x100?
| tests/python/CMakeLists.txt | ||
|---|---|---|
| 713 | There will be more ;) | |
| tests/python/sequencer_render_tests.py | ||
| 55 | Remove dead code. | |
Comment Actions
My bad - will change this.
Is there a conceptual difference between 480x270 tests and 200x100?
Now looking at code, there is not much of a difference, it's just when I was writing versioning, I had to keep in mind that scale to fit has to be taken into acount, but image_aspect / project_aspect are used only for scale, so it looks relatively safe.
So on one hand, we can expect only one case to fail. On the other hand, this is one of areas with a lot of complexity(proxy size has to be accounted for + other possible optimizations in future) and number of test cases could increase exponentially.
Comment Actions
Was testing the ctest -R sequence here with release build on the E5-2699v4 CPU. It takes in average 4 seconds for 12 reference images. Sounds a bit scary from scalability point of view (as in, adding more tests will make the test suit uncofortably slow).
How is your timing compares to mine? Did you try using smaller resolution (it seems that resolution can be dropped by a factor of 10)?
Comment Actions
On my system test took 1.83 sec. So I wasn't concerned too much.
I am not sure if I can lower resolution by factor of 10, because I will need to set some diff threshold after I will change scaling method, because it may not be pixel perfect. I don't know what the threshold will be though. Probably not very much. I will check this, also maybe PNG is not most optimal format at least for source images.
My point with scalability was rather that this is problem in general. Here it would be best to design components to be individually testable, and test in C++ I would say. I am personally quite skeptical about testing (tests are made by humans), but also I don't have any experience in this area, so let's say I was thinking out loud.
Comment Actions
You are likely to have higher clockrate than me. I'm not too much concerned about the timing at this point. Think it is acceptable. Way better than not having tests.
@Brecht Van Lommel (brecht), want to have a second thought here?
Comment Actions
Regarding the test files, the goal should be to have a test for all strips types and other important functionality. With that in mind, I wouldn't add 12 tests for crop and offset with different image resolutions. I'd image more something like one test for offset, one for crop, and one combining a bunch of different transforms to test transform order. And maybe blend two input images with different resolution in a single .blend.
The images should also have detail, colors and alpha. That way you can test that transform is preserving them correctly.
If you were writing unit tests you try to isolate things as much as possible, but with integration tests like this combining a bunch of things within a single test helps reveal real world issues and keeps the number of test .blend files under control.
I'm not sure I understand this. If you change the scaling algorithm, you need to update the reference images, not set a diff threshold. If the input images are too large to notice difference in the reference renders, you can make smaller input images.