Based on discussion with @billreynish and D5493, T68498.
The logic that combines toggling edit-mode with selecting isn't very helpful.
This patch uses a modifier key (Alt) to activate items.
This has the disadvantage that it wont work for middle mouse emulation, however Ctrl & Shift are already used (in the GSOC outliner branch).
----
As an alternative we could make this work by only clicking on the icons for example, or have a column for this purpose - see {T68498}.