Blenderartists thread with screenshots and descriptions.
http://blenderartists.org/forum/showthread.php?369218-GPU-GTX-580-Cycles-Rendering-Branched-Path-Tracing-CUDA-error-out-of-memory

When I was doing some more tests, I noticed that Branched Path Tracing uses much more vRAM than Normal Path Tracing. Is that normal?

2.69 Path Tracing: 372MB vRAM:

2.69 Branched Path Tracing: 751MB vRAM:

2.74 Path Tracing: 848MB vRAM:

2.74 Branched Path Tracing: 1263MB vRAM:

I managed to render the scene in Blender 2.74 by unplugging any monitor cable from GTX 580 - using it only for rendering.

Yes, it is normal that branched can use more VRAM. It's different code path in the kernel which requires different memory per-gpu thread.

Some optimization happened during this release cycle, so you might want to have a look into a newer build: https://builder.blender.org/download/

I'll have a closer look into the report this week, but it's probably just increased VRAM usage caused by more features enabled in the GPU kernel which is the only way to solve by spliiting the kernel (work on this is in progress actually).

Thank You for the explanation of the problem, I'll try the latest build of Blender and see what will happen. I only wonder about that on 2 GTS cards 1GB VRAM the scene rendered OK but they have CUDA Compute capability 2.1, GTX 580 has 2.0. Thank You once again :)