First time submitting a patch for the addons repo, let me know if I did it right.
As pointed out in the comments of the task (T76772#932038) this was caused by the changes to the Track To defaults, so I added @Julian Eisel (Severin) as reviewer for this as punishment.
On a side note: I checked all other cases of "TRACK_TO" in the addons repo to see if there are any other addons that might have this issue, but in all other cases, the track axis and up axis are always specified after creating the constraint. Lucky!