Turned out one video was enough to get severe stuttering in the timeline. Duplicating the file and creating that many objects each with its own unique material does indeed make the situation worse. I was not however able to reproduce the F12 freeze in this test for now.

The low playback performance is comparatively a minor issue so i decided to remove it from the task. Maybe i'll open another task instead.