An interesting Twitter thread sprung up recently discussing the merits of Tesla’s AutoPilot partially automated driving system relative to competitors. This came in the wake of a preliminary report from the National Transportation Safety Board that was examining a March 2018 crash that killed an Apple engineer in a Tesla Model X. One of the commenters raised the concept of which automated driving systems are most developed, citing how far back Tesla ranked in Navigant Research’s Automated Driving Leaderboard report.

How to Interpret Leaderboard Rankings

While it is true that Tesla had the lowest score in this year’s ranking, it’s important to understand both what is being ranked and how most developed may relate to best developed for automated driving.

This Navigant Research Leaderboard is intended to provide a snapshot of where analysts believe companies rank in their ability to successfully commercialize automated driving technology. To do that, the Leaderboard goes well beyond just the core automation technology, with individual subscores assigned for 10 different criteria including vision, go-to market strategies, partnerships, manufacturing capability, product quality, and financial strength. While Tesla has always ranked high on vision, it often lags in many other areas, leading to its low overall score.

Scoring for Vision

The vision score is based in part on where a company sees its business going in the coming decade as well as what it hopes to achieve. In Tesla’s case, its score was boosted by its belief that automated driving technology should be deployed as far and wide as possible as soon as possible.

That is a fantastic idea. In an ideal world, it would save many of the tens of thousands of lives lost annually in American traffic accidents and the more than one million lost globally. Unfortunately, we don’t live in an ideal world. Putting insufficiently developed technology in the hands of untrained consumers on public roads can be a recipe for disaster.

The Gap between Most and Best

AutoPilot may indeed be the most developed system on the market today in terms of the manufacturer’s willingness to stretch its capabilities and extract all it can from the available sensor suite. That does not necessarily make it the best developed system in terms of what should be in production based on safety requirements and consumers’ current abilities and understanding of the technology.

Humans Like Consistency

Early in my engineering career, working on anti-lock braking systems, I learned the value of ensuring that the technology performed with as much consistency and predictability as possible. For all our flaws, humans are remarkably adaptable if we understand the conditions. Most people tend to drive the same vehicle every day for years. Even if a vehicle has limitations or quirks, as long as they are consistent, drivers will adjust how they use the vehicle. If a vehicle has a longer stopping distance, drivers will brake sooner. If the steering response is slower, a larger angle will be used.

The problem comes when the system responds differently every time you use it, as many voice control systems have done. That’s when people stop using it or get caught unawares when they do. That likely contributed to Walter Huang ignoring warnings to hold the steering wheel in his Tesla. There are other systems on the market, that while not as capable as they could be in certain conditions, are indeed better developed for the real world.

Acknowledging Room for Improvement

As we move to higher levels of automated driving in the next several years, we need to encourage manufacturers to acknowledge what their products can’t do yet, while working to make available functions the best they can be.