Approach for a Reliability Proof of Autonomous Vehicles Compared to Human Drivers Through Real-World Road Tests

Real-world testing, along with extensive simulation, is essential for validating autonomous vehicle performance under real operating conditions. However, proving the reliability of autonomous vehicles compared to human drivers through real-world road tests has been challenging due to the lack of a comparable database on human driving behavior within similar Operational Design Domains (ODDs). A study by UMTRI collected human ride hail data, including crash statistics, in San Francisco from 2016 to 2018, enabling a comparison between autonomous vehicles and human driving behavior over several years. The approach presented in this report evaluates the reliability of autonomous vehicles based on real-world tests on public roads, using data from Waymo autonomous vehicles operating as robotaxis in San Francisco. An analysis of disengagement data revealed improvements in later developmental stages for disengagements due to software discrepancies. The reliability of Waymo driverless vehicles in San Francisco was demonstrated and compared to human drivers from ride-hail services in the same ODD. Based on DMV data as of February 2024, it was shown that Waymo driverless vehicles in San Francisco are at least as reliable as human drivers with similar driving behavior in the same ODD. Additionally, even under conservative considerations, the reliability of human drivers was proven with a high confidence level.

Keywords: Reliability proof, Autonomous vehicles, Real-world testing, RGA (Reliability Growth Analysis), SRGM (Software Reliability Growth Model), Disengagements, Accidents with AV involved, Comparable ODD.