I don't think you can do it with just any layers, because some aren't "applicable" to the rule.
So, you can't do Board Outline Clearance to 3DComponentBody, or Component Clearance to IsBoardOutline or IsKeepOut.
This seems like an ideal application for Rooms. Set placement for within rooms only, and set the room outline to the desired shape.
Or, place 3D bodies around the outline, representing the v-score tool keepout areas. Hacky, but it will do full 3D component clearance that way.
Tim