Moved from Patches to Blender 2.6 Bug Tracker

Freestyle does not account for layer masks at the moment. This is a known limitation of the present Freestyle implementation as stated in the Freestyle section of the Blender online manual.

The key issue preventing the support for layer masks from being implemented is that Freestyle strokes are 2D elements that do not have Z depth information. It is noted that layer masks rely on Z depth information.

In Freestyle, feature edges can be connected regardless of objects to which the edges belong. Just as an example, I have attached a Freestyle render showing rough strokes along external contours. The lines are actually three rounds of one stroke (having variable color from red to yellow) constructed by connecting non-occluding contour lines of three cubes. For this kind of strokes, it is not trivial to define Z depth along strokes. An obvious solution is to inherit the Z depth information of the original feature edges (the non-occluding contour lines in this case), but composite results based on such Z depth information are likely unsatisfactory and difficult to tune by users.

Another aspect that makes the problem more complicated is the orientation of stroke faces. At the moment stroke faces are parallel to the 2D image plane, but artists may want to have different stroke face orientations depending on the original mesh geometry, specific artwork objectives, and so on. In this case, just inheriting the Z depth information of the original feature edges would not be sufficient any more and direct user intervention would be necessary.

To me addressing the reported issue looks a major challenge. For this reason I would close the report for now.

When I faced this problem, my solution was to include both layers for the rendering, and filter the needed objects with an object group in the freestyle line set.
On the programming side, implementing layer masks could work the same way: making a union of the layers and mask layers, render it as usual, but filter out all objects not in the render layers just the same way as the object group filtering works.

Would be great if there were an elegant solution to this. Becomes even more difficult with partly occluded objects. I'm trying to hack together compositing solutions with multiple layers, But it's a hacky and imprecise way to go. Good luck!