The first reports on self-driving road tests in China are out, and they suggest that people who want to know how good China’s autonomous technology is would be better off turning to California.
Beijing became the first city in China to allow testing on public roads in late 2017, and last year 56 vehicles from eight companies logged more than 150,000 kilometers (93,200 miles) of testing in the capital, according to one of the reports (link in Chinese), which Beijing’s Municipal Commission of Transport put out on the weekend. A separate report by the Beijing Innovation Center for Mobility Intelligent (link in Chinese), a government-affiliated research center, included more information. Based on data it got from monitors installed in self-driving cars, it described 23 situations that led to autonomous vehicle disengagements, requiring humans to take over.
But neither report detailed how frequently humans had to intervene per kilometer driven, a flawed but important metric for measuring just how autonomous the technology is.
Baidu, which has been developing its Apollo autonomous driving platform, tested 45 of the total vehicles, which accounted for nearly 140,000 km (87,000 miles), or more than 90% of the total mileage. That was followed by Pony.ai, a three-year-old startup founded by two former top engineers at Baidu, which logged around 10,000 km (6,200 miles) with two vehicles. Shanghai-based electric vehicle startup NIO came third. Ride-hailing giant Didi was last, clocking less than 80 kilometers with a single vehicle. The cars drove on 44 public roads, covering 123 km, according to the reports.
Still, we’re not entirely in the dark about either Baidu’s or Pony.ai’s disengagement rates. Both companies are also testing their cars in California, which requires firms to disclose certain information related to tests on public roads, such as disengagements, which is then made public every year.
Data for 2018 shows that Baidu’s cars traveled 29,118 km (18,000 miles) on California roads, and saw a disengagement once every 331 kilometers on average, according to Nikkei. That’s an improvement from 2017, when disengagements happened roughly every 27 kilometers in the first half of the year, reducing to once every 191 kilometers in the second half. If its cars disengaged at a similar rate in Beijing as they do in California, Baidu would have seen more than 400 disengagements in its Beijing tests.
Pony.ai performed better than Baidu in California, clocking 26,322 km (16,355 miles), according to Nikkei, and registering one disengagement only every 1645 kilometers.
Meanwhile, Waymo, the self-driving subsidiary of Google’s parent company Alphabet, tops the California list—it did more than 2 million km (1.3 million miles) last year, and recorded disengagements only once every 17,731 kilometers.
Baidu declined to comment. Pony.ai didn’t immediately respond to questions.
Driving conditions, of course, are different in Beijing and it’s possible both companies recorded different disengagement rates in China’s capital. Unpredictable driving by other vehicles—which Quartz observed first-hand last year in a Pony.ai vehicle—might make disengagements a more frequent occurrence, while other differences in the testing process might have the opposite effect.
The BICMI report described a number of situations involving improper driving by other vehicles that led to disengagement, without providing specifics for any company.
They happened, for instance, when other cars didn’t stay in their lane and barged into the lane being used by the autonomous vehicle; when other cars were going the wrong way; when cars pulled over by the side of the road unexpectedly—as might happen when a taxi stops to drop off a passenger; and when people rode bicycles on a single-lane road blocking the self-driving car.
Since May 2018, China has allowed all provincial and city-level governments to issue road-test licenses as part of the country’s aspirations to develop what it calls “intelligent vehicles.”