System Information
iMac late 2013, 16GB Ram, OSX Sierra (10.12.1), Nvidia GForce GTX 775M 2GB

Blender Version
Broken: v2.78a (downloaded dec 10, 2016) with Experimental features enabled
Worked: v2.78a (same download) with Experimental features disabled

Blender will preview-render this scene just fine, it does not render on hitting F12. Instead, Blender returns an out-of-memory error.

When I hit F12, Cycles goes into lockdown while it is tesselating. After about a minute, it comes up again with "out of memory error", with peak memory usage at about 0.05Mb.

When I turn off the new feature, there's no problem either previewing or rendering

Blend file available here: https://www.dropbox.com/s/7tb31dm85hmw4rg/Hex%20Planet-tesselationerror.blend?dl=0

Exact steps for others to reproduce the error

• Open Blender 2.78a (latest official) OSX version
• Switch on Experimental Features in Render tab
• Open Hex Planet-tesselationerror.blend
• Hit SHIFT-Z -> preview render works as expected (Hit SHIFT-Z again to disable render preview)
• Hit F12 -> Tesselation and BVH build are very slow. After a while the Out of Memory error occurs when Blender eats the available application memory
• Switch off Experimental Features in Render tab
• Hit F12 -> Render works fine with the non-adaptive subsurf modifier