If you let the auto drive without human intervention for 99.9% of the time, the human won't be paying attention for the 0.1% ... it's as simple as that.
Yep.
The only way fully automated vehicles, without any human driver, will become a full replacement and even achieve better safety figures overall than human-driven vehicles (which is what they promise us), IMO, is if ALL the infrastructures become exactly adapted to them and there is no human-driven vehicle (or pedestrian, or ...) on the roads anymore. Not a single one. (I think it's been said already in some above posts.) Even though bugs are always possible (and even relatively common), machines do operate much better with other machines than with humans or anything unpredictable when it comes to safety.
Once we get there, most of the "AI" that we're currently working on for this will become pointless. If the only vehicles in circulation are all interconnected and all with compatible routing characteristics, avoidance will be a much simpler task.
And I kind of think at this point that this is exactly where people/companies working on and advocating autonomous vehicles are leading us to. A world, in fact, where only machines will be allowed, in the end. We will be just passengers, until we are not even needed anymore (that last part is a bit sci-fi, but the fictitious character is debatable.) The sophisticated obstacle avoidance AI stuff (and dealing with uncertainty in general) is entertaining a lot of people these days, but I'm not sure this has any future as is.