Configuration file cycles-x/config.py:

devices = ['OPTIX']
categories = ['cycles']
builds = {
  'cycles-x': '/home/brecht/dev/build_linux/bin/blender',
  'master': '/home/brecht/dev/worktree_build/bin/blender',
}

Terminal output:

$ ./benchmark run cycles-x
barbershop_interior                      cycles-x                       7.5451s
barbershop_interior                      master                         40.3464s
bmw27                                    cycles-x                       8.3284s
bmw27                                    master                         10.2636s
classroom                                cycles-x                       11.4427s
classroom                                master                         25.2872s

Graph:

Nice work, and great to see the "in-house" tools are becoming more available for all developers.

The configuration file seems tricky. What is going to happen if one have multiple OptiX-capable devices?

For the CI type of things, you also don't want to maintain the configuration on every machine. Is it possible to configure the script to get benchmark on every available compute device?

For multiple devices, I made it work like this now.

$ ./benchmark list
DEVICES
AMD Ryzen Threadripper 2990WX 32-Core Processor (Linux)            CPU
NVIDIA RTX A6000 (Linux)                                           CUDA_0
NVIDIA RTX A6000 (Linux)                                           OPTIX_0

TESTS
...

And then in the configuration you could for example have:

devices = ['CPU']
devices = ['OPTIX_0']
devices = ['OPTIX_1']
devices = ['OPTIX_*']
devices = ['CPU', 'OPTIX_*', 'CUDA_*']
devices = ['*']

Test and category names support similar wildcards already.

From the feature creep department: Running the benchmarks on regular intervals is great, but to make the data a little more useful something more may be needed, @Dalai Felinto (dfelinto) already has a grafana+influxdb instance setup at https://metrics.blender.org/ . Influxdb is incredibly easy to work with, posting data is usually nothing more than a curl command. (but a python lib is available if required)

That being said, I don't think not having that integration should stop this from landing

InfluxDB + Grafana are quite a big ask for a developer to set up locally, just to generate some graphs. The JSON + Google Charts used now requires no setup, but is not as powerful. I'm not sure if we should support just one, or both.

There are also some complexities:

• Devices, test categories and tests will change over time. The number of graphs shown needs to dynamically adapt to that.
• Sometimes you need to remove a bad run, or drop old results because some change in Blender or tests files makes comparison before/after invalid.
• The old implementation supported running tests multiple times and displaying error bars, I would like to restore that.
• There also used to be support for showing the rendered image for each test/revision on the web page. Probably will not restore that, but maybe.

However we do this, probably best to figure out how to integrate this with CI and potentially InfluxDB + Grafana as a separate step.

@Brecht Van Lommel (brecht), Lovely!

For the Graphana story: is all interesting and something we should investigate. The way I was always imagining this is to have a "core" of benchmark shared across different end-points, so that we can ensure developers and CI metrics are always calculated the same way. But one thing to keep in mind: simple things should be simple, complicated things should be possible. For the local development it should be possible to easily setup the benchmark and use it, without extra dependencies.

@Brecht Van Lommel (brecht), Unfortunately, didn't have time to apply it and test, but am i right the script can already replace the benchmark scripts we are using for Cycles X? If so, let's make all the effort of pushing it to the "production" :)

For the review: on a top-level it seems fine, but some local aspects makes me a bit sad :( For example, we shouldn't mix old-style os.path with the more modern/easier pathlib. Not sure this is something you've been planning to do still, or want some help with addressing things like that.

@Brecht Van Lommel (brecht), Unfortunately, didn't have time to apply it and test, but am i right the script can already replace the benchmark scripts we are using for Cycles X? If so, let's make all the effort of pushing it to the "production" :)

It can replace the scripts, but I had to locally modify the contents of the lib/benchmarks folder to place the cycles-x .blend files there. That is the main thing to address before committing I think.

What we could do is:

• Commit this to master, with a simple .blend file load timing test using the existing lib/benchmarks/cycles files. Then other developers will be able to add tests for animation playback, mesh editing, etc.
• Create a lib/benchmarks/cycles-x folder with the new cycles-x test files.
• In the cycles-x branch, enable benchmarking with those new files.
• When merging to master, update lib/benchmarks/cycles to have a single set of .blend files and remove the cycles-x folder.

Does that seem reasonable? I'm unsure if it would be fine to continue committing more big files there, like a couple gigabytes of production files, or if that will put too much strain on the svn server.

For the review: on a top-level it seems fine, but some local aspects makes me a bit sad :( For example, we shouldn't mix old-style os.path with the more modern/easier pathlib. Not sure this is something you've been planning to do still, or want some help with addressing things like that.

I removed the usage of os.path now, just a few places I forgot. Along with other cleanups, it's ready to be reviewed at that level now.

Does that seem reasonable?

Sounds perfectly fine to me.

For the big files concern: I am not sure it will be different for SVN compared to how the libraries are handled. I don't think we should have that much of production files, as I don't think adding more of them doesn't really mean improved quality of benchmarking. From the disk usage on the server point of view, it wouldn't really matter that much if something is on "ftp" folder or in SVN. The possible downside is that with SVN there will be an "extra" copy of the files in the ".svn" folder. But all factors combined, I'd say we should just stick to SVN: just makes it easier to (a) setup the environment (b) make sure everything is up to date.

I removed the usage of os.path now, just a few places I forgot. Along with other cleanups, it's ready to be reviewed at that level now.

Cool. Left couple of inlined notes.

Not really a python person, but isn't it preferred to use typed classes instead of dicts (could be so that Entry should become own class) ?

Again, just on a note level. Don't really think those are stoppers or anything.

I'm almost tempted to say that we should go ahead and commit it, and solve possible issues later. Is not particularly fun to apply patches on the SVN checkout :)

LGTM for master. Will perform a test run here.

In D11662#300562, @Sergey Sharybin (sergey) wrote:

Not really a python person, but isn't it preferred to use typed classes instead of dicts (could be so that Entry should become own class) ?

It was a dict to match JSON, but converting it to a dataclass is easy so I did that.