When I talked to @Sebastian Parborg (zeddb), he said that the update rate of the grab brush was way too high and it is causing performance issues. I limited the updates per second with a timer and the performance increases drastically
Current master:
This patch:
The 0.07 value was obtained experimentally on my computer, but it is probably not optimal. I'm not sure if this is the right way to do this.