The biggest problem I see with driverless cars are not with these cars, but those that are still being driven by Humans... there are so many bad drivers on the road that it really only takes one bad one to screw it up for everyone else.
http://www.techspot.com/news/53158-driverless-cars-to-have-a-tremendous-impact-on-our-cities.html