I believe the Keep-Out Layer is absolute; you can set distance to it, all the way down to zero, but any overlap is absolutely prohibited.
This appears to be one downside of using KO for board outlines/cutouts. Consider moving these graphics to a mechanical layer, or drill drawing?
As of AD 16.1 at least, the Board Outline Clearance rule class (I forget if it was in AD15 as well?) seems to work, but it's... kinda weird? It doesn't show violations (maybe because the board outline isn't a real object and it needs two objects to point at and say "aha"?), and it works during interactive routing (but the collision outline isn't shown with CTRL+W*) but not when dragging traces for example.
*Which I think was added AD16 so don't mind that.
Oh, in AD18 or so (or maybe it was 20, I forget?), they revamped the keepout system, I wonder if that affects this particular situation. Didn't think to try it...
Tim