Ignore most of the cmake hackery, yes it's kludgy and there's bits of code duplicated here and there, i'll clean that up at a later point in time, best not to be distracted by this for the rest of the review.

Naming: Is "descriptor" too much similar to vulkans descriptors set? This might create confusion. Maybe MetaData is better? But it may hold more than meta data.

Yes descriptors have a different meaning in vulkan. "A descriptor is an opaque data structure representing a shader respirce such as a buffer view, image view, sampler or combined image sampler".

We are referring to shader bindings and stage interfaces, what could also be considered as a binding. BindingInfo?

Choose the right extension for descriptor files.

Should reflect the term we chose for descriptors.

Choose the right folder. Do we keep them alongside shader files even if name could differ?

Prefer to keep it close to the shader files. although as they are more generic and shaders could be used by multiple descriptors We could move it to a folder next to shaders.
/shaders/*.glsl
/shader_desc/...

Not to far away, but also not seen as the same category of files.

Indentation is 4 spaces for some reasons in descriptors definitions.

Could be that your IDE detects a different format and use other settings. Would prefer to keep it to 2 spaces for readability.

Memory usage might be a concern. We are talking about ~3 MByte of data in the executable.

Would expect that most would be for the workbench variations. We could see how we can add the concept of variations into the descriptor so we could reduce memory usage but with a bit of overhead. We can do some profiling to see where the memory goes to.

If memory is a concern we could also add a mechanism to have active an inactive descriptors. inactive could be offloaded/compressed.

Syntax: Is using [0] to specify binding index clear enough or using VERTEX_INPUT(0, ...) makes more sense? What about future proofing if we one day want to compile this as C++ code.

I would prefer VERTEX_INPUT. it is hard to know what issues there will be in the future. having a macro here would add some flexibility.

This format has nice benefits but it needs manual writting on our ends to modify the shaders. That said, writing all shaders variations for the workbench prepass was quite easy and fast.

Other benefit is to generate the shader documentation for gpu module for builtin shaders. They are out of date.

How would we expose this through pyGPU?

I would suggest a must have for writing custom shaders. But would also be nice to traverse it when a shader is built (for debugging/life coding).

It's not really clear to me why this uses code generation for the GPUShaderDescriptor data structure. Is there some reason this can only be done at compile time, or is it an optimization to avoid runtime overhead? I understand there may be some slow operations like processing shader source code that have a good reason to be done at compile time, but would like to understand the motivation. If this is to be used for the Python gpu module also, I guess there must be a runtime implementation as well?

It might be useful to take inspiration from the new C++ node socket declaration code, if you wanted a single system that can both be quite readable as a declaration but could also be wrapped with a Python API.

b.add_input<decl::Float>(N_("Scale"))
    .min(-1000.0f)
    .max(1000.0f)
    .default_value(5.0f)
    .no_muted_links();

@Brecht Van Lommel (brecht)

It's not really clear to me why this uses code generation for the GPUShaderDescriptor data structure.

The main reason is that we can precompile many variations at build time and assign byte code / SpirV to every one of them. However, it is not the goal to generate the shader variations sources and write them. They will still be created at runtime.

Another reason is that it makes every GPUShaderDescriptor static so we can reference them from either C or C++ without interfaces nor setup code.
It also puts every descriptors in a single file and avoids any ordering conflict by doing the dependencies at compile time.
This is also a good way to validate every shaders at this point.

It might be useful to take inspiration from the new C++ node socket declaration code,

This does look nicer P2628. However this mean that declaring and defining is only doable in C++. Which might not be a real issue in practice since the only part that needs to manipulate GPUShaderDescriptiors is EEVEE materials which is (in eevee-rewrite) already written in C++.
I might find a way to only expose handles to the actual descriptors and define them at runtime. This would avoid the memory usage in the binary and keep the preprocessing simpler.

@Jeroen Bakker (jbakker)

We are referring to shader bindings and stage interfaces, what could also be considered as a binding. BindingInfo?

There is more than bindings inside this structure so I don't think it fits. Here are more propositions:
GPUShaderDescription
GPUShaderCreateInfo
GPUShaderMetaInfo
GPUShaderMetadata
GPUShaderInfo

If memory is a concern we could also add a mechanism to have active an inactive descriptors. inactive could be offloaded/compressed.

The memory concern is only about the size of the binary. They are likely not all loaded at runtime.

Prefer to keep it close to the shader files. although as they are more generic and shaders could be used by multiple descriptors We could move it to a folder next to shaders.

What about putting descriptors in workbench/shaders and source files in workbench/shaders/source or src.

Could be that your IDE detects a different format and use other settings. Would prefer to keep it to 2 spaces for readability.

It is not about my IDE but about clang-format. I don't think it has to do with the file extension. I could not find a way to make it work so I'm asking for help here.