An interesting Twitter thread sprung up recently discussing the merits of Tesla’s AutoPilot partially automated driving system relative to competitors. This came in the wake of a preliminary report from the National Transportation Safety Board that was examining a March 2018 crash that killed an Apple engineer in a Tesla Model X. One of the commenters raised the concept of which automated driving systems are most developed, citing how far back Tesla ranked in Navigant Research’s Automated Driving Leaderboard report.

How to Interpret Leaderboard Rankings

While it is true that Tesla had the lowest score in this year’s ranking, it’s important to understand both what is being ranked and how most developed may relate to best developed for automated driving.

This Navigant Research Leaderboard is intended to provide a snapshot of where analysts believe companies rank in their ability to successfully commercialize automated driving technology. To do that, the Leaderboard goes well beyond just the core automation technology, with individual subscores assigned for 10 different criteria including vision, go-to market strategies, partnerships, manufacturing capability, product quality, and financial strength. While Tesla has always ranked high on vision, it often lags in many other areas, leading to its low overall score.

Scoring for Vision

The vision score is based in part on where a company sees its business going in the coming decade as well as what it hopes to achieve. In Tesla’s case, its score was boosted by its belief that automated driving technology should be deployed as far and wide as possible as soon as possible.

That is a fantastic idea. In an ideal world, it would save many of the tens of thousands of lives lost annually in American traffic accidents and the more than one million lost globally. Unfortunately, we don’t live in an ideal world. Putting insufficiently developed technology in the hands of untrained consumers on public roads can be a recipe for disaster.

The Gap between Most and Best

AutoPilot may indeed be the most developed system on the market today in terms of the manufacturer’s willingness to stretch its capabilities and extract all it can from the available sensor suite. That does not necessarily make it the best developed system in terms of what should be in production based on safety requirements and consumers’ current abilities and understanding of the technology.

Humans Like Consistency

Early in my engineering career, working on anti-lock braking systems, I learned the value of ensuring that the technology performed with as much consistency and predictability as possible. For all our flaws, humans are remarkably adaptable if we understand the conditions. Most people tend to drive the same vehicle every day for years. Even if a vehicle has limitations or quirks, as long as they are consistent, drivers will adjust how they use the vehicle. If a vehicle has a longer stopping distance, drivers will brake sooner. If the steering response is slower, a larger angle will be used.

The problem comes when the system responds differently every time you use it, as many voice control systems have done. That’s when people stop using it or get caught unawares when they do. That likely contributed to Walter Huang ignoring warnings to hold the steering wheel in his Tesla. There are other systems on the market, that while not as capable as they could be in certain conditions, are indeed better developed for the real world.

Acknowledging Room for Improvement

As we move to higher levels of automated driving in the next several years, we need to encourage manufacturers to acknowledge what their products can’t do yet, while working to make available functions the best they can be.

If we’ve learned anything from the era of reality television and user-generated online video, it’s that a surprising number of people will risk great harm by misusing themselves or technology to get some online attention. Whether it’s blowing up a microwave, eating laundry detergent pods, or misusing driver assist features on a car, too many are willing to abandon common sense in search of the dopamine hit that comes with seeing the number of views ratchet higher. I shake my head in bewilderment when I hear of someone swallowing a detergent pod, but at least they are not putting others in harm’s way.

Vehicle Travel Should Be Serious

More concerning is seeing videos of people using today’s vehicle partial automation systems, like Tesla AutoPilot, beyond the scope of its capabilities or trying to figure out how to trick it into functioning as a more highly automated system. I have no issues with hardware hacking of stationary devices, or vehicle systems not related to driving. Repurposing hardware you have purchased to provide added functionality can be fun, educational, and allows you to extract more value from it.

But modifying or tricking a vehicle’s guidance system puts innocent bystanders at risk, with potentially disastrous consequences. People who override driver assistance systems or pay little attention to the vehicle’s operation could negatively affect the adoption of automated vehicles.

Consumers Shouldn’t Overestimate Vehicle Autopilot

Tesla AutoPilot and similar systems from General Motors, Volvo, Mercedes-Benz, Nissan, and others are not automated driving systems. Except for GM’s SuperCruise, none of these systems are reliably able to hold a vehicle in lane to the degree of hands-off functionality. All of the driving systems, including GM’s, require the driver to remain engaged with eyes on the road and ready to take over.

Overconfident Users Are Misusing Existing Automated Capabilities

While Tesla CEO Elon Musk often talks about software updates that will give AutoPilot full self-driving capability, that day has not arrived and may never be here with the current generation of hardware. Despite the well-known flaws and limitations of AutoPilot, Tesla owners continue to ignore warnings from the system and the company, using the system in ways or in places where it should be disabled. One owner that has posted dozens of videos to YouTube recently tried to demonstrate that stuffing oranges between the steering wheel rim and spokes could fool the system into thinking the driver’s hands were on the wheel. Had this been done on a closed track, it might have been an interesting stunt. On a public road, with other vehicles around, this was downright reckless.

An Apple engineer recently died when his Tesla was on AutoPilot mode and ran into a highway barrier in California. While the system clearly failed to hold the vehicle in the lane, this driver had previously complained about the car exhibiting the same bad behavior to Tesla service. Since the accident, several other Tesla owners have replicated the situation while recording video with a hand-held phone, risking further injuries.

A pedestrian was killed by an Uber autonomous test vehicle in another instance of a driver not paying attention as instructed and pushing the technology beyond its limits. Automakers need to continue clarifying the vast differences between the driver assist technologies of today and the driver not needed technologies of tomorrow.

Holding out Hope for Progress

A number of studies have already shown that a majority of people don’t trust automated driving systems. Automation has the potential to provide enormous societal benefits by saving lives and damage to property. However, if the actions of those looking for views erode public trust in the technology even as it improves, those benefits may remain off in the horizon.

For many of us that keep tabs on the automotive industry for a living, the first 2 weeks of January are among the most grueling of the year. The North American International Auto Show in Detroit has kicked off the year for several decades. And in the past 10 years, International CES in Las Vegas has become an increasingly important addition to our schedule as the two events run back to back. The announcements at 2018’s shows illustrated some of the crucial interconnections between the growth of technology and the transportation business.

For automakers, CES has largely been a place where they talk about future technologies and try to shift the media’s perception of them from being old-fashioned metal benders to forward-thinking visionaries. They rarely show actual new products, instead focusing on automated and connected concept vehicles. The Detroit show, like most other auto shows, targets consumers that are buying vehicles in the coming year.

For an industry that is facing the biggest transformation in more than 100 years, this is a crucial time. While many recent auto shows have highlighted new plug-in and hybrid vehicles, there were almost none in Detroit this year. Instead, the biggest announcements came from the Detroit-area manufacturers, and they were all pickup trucks—mostly full-size. Fiat Chrysler unveiled the redesigned 2019 Ram 1500. Chevrolet brought out a new from the ground up Silverado, and Ford launched a diesel version of the F-150 and a midsize Ranger pickup.

Profit in Pickups

Pickups are a segment that is likely to be among the last to gain highly automated driving capabilities, as discussed in Navigant Research’s Market Data: Automated Driving Vehicles forecast and its Leaderboard reports. However, those automation technologies were a major topic of conversation in Las Vegas, particularly in the context of whether manufacturers will build new business models around these costly, complicated, support-intensive vehicles.

That’s why pickups are so important to Detroit. They are the profit engines that keep this industry humming along while indirectly funding R&D efforts that will create the next big things. Part of why Ford is bringing the Ranger back to North America is that the average selling price of an F-150 is now more than $58,000. Pickups and large SUVs generate far more profit per vehicle than any small car and they sell in far larger volumes than any other segment in the American market. Ford is projected to make a full-year 2017 profit of more than $9 billion, largely thanks to sales of nearly 900,000 F-series trucks. Even the third place Fiat Chrysler sold more than 500,000 Ram pickups in 2017.

All three manufacturers are adopting fuel efficiency technologies such as 48 V mild-hybrids, dynamic cylinder deactivation, diesel and active aerodynamics in order to meet fuel economy requirements, as discussed in Navigant Research’s Automotive Fuel Efficiency Strategies report. However, until they all figure out how to make sustainable profits in the new age of mobility, we can rest assured that they will continue pressing ahead with enhancing the customer appeal of these trucks in order to keep the cash flowing to develop the promises made at CES.

Let’s be absolutely clear about something. As I write these words in October 2017, there are exactly zero self-driving vehicles available for consumers to purchase in America. In fact, Elon Musk’s proclamations and pre-sales of non-existent technology aside, it will likely be at least several more years before an individual can buy a self-driving vehicle. With that in mind, the media needs to stop using the term self-driving in the context of any production vehicle.

Misleading Headlines

In recent weeks, Cadillac has conducted a cross-country media preview of the 2018 CT6 sedan with the first production application of its Super Cruise system. From the event launch in Manhattan to its arrival in Los Angeles 2 weeks later, media outlets including NBC’s Today show, USA Today, Business Insider, and Fortune have referred to Super Cruise as self-driving. In doing so, they are doing a disservice to their own credibility, to consumers, to General Motors (GM), and to every engineer working on automated driving technology.

To its credit, GM itself never calls this self-driving or automated technology. Following the 2014 ignition switch recall, GM instituted new safety review procedures on new products and those changes are reflected in the capabilities and limitations of Super Cruise.

What Is the 2018 CT6 Super Cruise?

This is a very capable advanced driver assistance system (ADAS) similar in principle to Tesla Auto Pilot, Volvo Pilot Assist, and Mercedes-Benz Drive Pilot. Navigant Research’s Automated Driving Vehicle Technology report projects that these types of systems—defined as Level 2 partial automation by SAE—will account for nearly 59 million sales annually by 2026.

I personally spent nearly 900 miles with the system over 2 days. Within its operating domain, it works very well. But the key is that operating domain or definition of where the system can work. This is a supervised partially automated assisted driving system. On divided highways where there are no intersections, cyclists, or pedestrians, Super Cruise can handle steering, acceleration, and braking with the driver taking their hands and feet off the steering wheel and pedals.

That doesn’t make it automated. As the Super Cruise branding implies, this is a more advanced adaptive cruise control. The driver must still watch the road and be ready to take over when the system encounters a situation it cannot handle such as a construction zone, lane merge, or faded lane markings.

A face tracking camera similar in principle to the Face ID system on the upcoming Apple iPhone X watches for facial and eye movements to ensure the driver is alert and attentive, something no other current ADAS does. Meanwhile, high definition navigation maps prevent inappropriate use on surface streets.

Misrepresentation Leads to Unrealistic Expectations

By continuing to call Super Cruise self-driving, media creates unjustified expectations of its capabilities with consumers. GM’s design approach should reduce the sort of misadventures we’ve seen from Tesla drivers on YouTube. However, customers that bought into the media hyperbole may be disappointed with the more cautious assisted driving technologies. Even as the technologies become more sophisticated, customers burned by misleading headlines today may remain skeptical and decide to hold off on future purchases, which damages the overall goal of improving safety on the road.

Patience

It’s one thing to take someone like Elon Musk at his word and assume he’s going to deliver what he promises (which he often eventually does, albeit over budget and months to years late). But by inaccurately portraying what a product can and more importantly cannot do, customer interest (and possibly safety) can be compromised. Truly automated vehicles will get here soon enough. Let’s not rush to mislabel.

Over the past several years, there has been an ongoing narrative that a battle has sprung up between Silicon Valley and the auto industry. The tech industry hype machine wants the world to believe that venture capital-backed startups are going to appear with some magic technology that disrupts and destroys the century-old incumbents. The reality is likely to turn out quite differently, with some of the brightest minds in the valley coming up with cool ideas that become a key part of the transportation ecosystem.

Tech Has Saved the Automobile Industry Before

The fact that the auto industry has remained vibrant over the past 50 years can in large part be traced to innovations that have emerged from the San Francisco Bay Area, particularly the silicon microprocessor that gave the region its nickname. At the onset of environmental regulation at the end of the 1960s, most of the functional aspects of cars were mechanically controlled, and these vehicles consumed more fuel and spewed more pollution than they do today.

As engineers struggled to meet the new regulatory requirements, the industry entered what became known to car enthusiasts like myself as the malaise era. Attempts to better control engines through mechanical means like vacuum lines led to many terrible engines with weak output, awful drivability, and barely improved emissions and efficiency.

Silicon Valley saved the auto industry from being suffocated by regulations. As early microprocessors and sensors were applied to engine and transmission management as well as new safety systems like anti-lock brakes, it became clear that computers in the car would be the key to enhanced driving. By the mid-1980s, electronic controls were enabling engineers to extract more power while using less fuel and cleaning up emissions. As fuel economy regulations stopped climbing, car companies offered customers improved performance and capability without making them spend more at the pump.

After earning my degree in mechanical engineering, I spent the next 17 years working on improving vehicles through  more sophisticated software running on a series of cheaper, yet more powerful slivers of silicon. Today’s most sophisticated vehicles utilize anywhere from 50 to 100 onboard computers to manage everything from lights that follow the angle of the steering wheel to automatically maneuvering a truck to connect a trailer.

Looking Forward to More Industry Collaboration

Silicon Valley has been a key enabler of the modern vehicle for decades. As we shift toward a world where most of the driving is done by software instead of people, the tech and auto industries must continue to collaborate more closely. The auto industry has developed an immense base of knowledge in building complex pieces of hardware at high volume and with high degrees of reliability and durability. Those machines come in a huge variety of configurations to meet virtually every possible transportation need.

Meanwhile, the tech industry has an unrivaled set of capabilities in developing software and electronics and driving down costs while improving performance. There are great minds on both sides focused on how to make mobility safer, cheaper, and more universally accessible. The Navigant Research Leaderboard Report: Automated Driving scored automakers and tech companies on their likelihood of success in commercializing this technology.

Almost everyone recognizes that transportation will change in the coming decades. The collaboration between the tech and auto industries has yielded incredible results for nearly half a century. New partnerships are going to form on the way to fully automated driving. There’s no need to spin those relationships into a competition when greater collaboration will likely yield much better results for everyone moving forward.

An old axiom in motorsports goes: “to finish first, first you must finish.” This means you can have the fastest car on the track and qualify on the pole position, but if you don’t have the preparation or team to back you up, the quality of the car is meaningless. In the race to make automated driving a successful commercial reality, hype may get companies all the media attention, but a fully realized strategy combined with the ability to execute are the keys to success. This is why Ford, General Motors (GM), the Renault-Nissan Alliance, and Daimler are the leaders in the latest Navigant Research Leaderboard Report: Automated Driving Systems.

Outside observers would not be faulted for believing that companies in Silicon Valley were about to roll over the entire automotive industry and take over personal mobility in the coming months based on news coverage. However, as many veterans of the technology industry have become painfully aware of, the reality is that building vehicles to safely transport the world’s population is far more difficult than just writing an app and publishing it to an online store.

Horse Before the Cart

Assembling a suite of sensors and writing the basic software to control a vehicle are actually the easy parts. Before that package can become a real product, you need a vehicle. Google developed its automated driving system in 2009 by hiring many of the top brains from Stanford, Carnegie Mellon, and several automakers that had previously created winning vehicles in the DARPA Grand Challenge program between 2004 and 2007. Then Google went to local Toyota and Lexus dealers and bought vehicles one or two at a time. Companies like Cruise Automation and Uber followed similar paths. In order to commercialize a system, they will need to invest billions more to develop and manufacture vehicles or find an automaker partner willing to supply cars.

Uber is reported to have lost more than $3 billion in 2016 without capital investment in vehicles or manufacturing. The world’s major automakers already have the engineering and manufacturing infrastructure in place, and many of them have been working on autonomous technology for far longer than Silicon Valley. Major automakers understand the intricacies of developing, validating, and certifying vehicles for profitable production.

At Navigant Research, we believe the leading automakers are learning what it takes to develop automated vehicles faster than new entrants can learn how to build cars. Companies like Ford, GM, Nissan, and Daimler also understand the regulatory and product liability hurdles faced by bringing automation to the world’s roads. These companies have heavily invested in controlling and understanding the key technologies required to make vehicles and automated driving system work seamlessly.

Just Around the Corner

The leading companies in this field are also rapidly developing their own in-house mobility services so that they can provide consumer access to automated driving systems while retaining control of vehicle manufacturing. This will help to ensure that the vehicles are properly maintained and updated—something that is key to safe and proper use within the early years of deployment.

Technology companies like Waymo and nuTonomy, as well as suppliers like Delphi and ZF, will have an important role to play in the new mobility ecosystem. But for now, automakers lead in the automated driving system race.

Back in September 2014 as the ITS World Congress gathered in Detroit, General Motors (GM) CEO Mary Barra announced that in 2016, a new Cadillac model would become available with the semi-autonomous Super Cruise system. With only a handful of weeks left in 2016, we now know that the Super Cruise will debut on Cadillac’s flagship CT6 sedan, but it won’t be arriving until sometime in 2017.

A lot has happened since that announcement, and GM has put a much greater emphasis on ensuring safety as a result of the massive ignition switch recall that began early in 2014. Those process changes have led to some significant upgrades to Super Cruise in an effort to avoid the issues caused by human interactions with Tesla’s similar AutoPilot driver assist system. Navigant Research’s Autonomous Vehicles report projects that by 2020, approximately 13 million vehicles with these so-called Level 2 automation systems will be sold annually.

Geofencing

In the process of evaluating the safety of Super Cruise, one of the key differences that GM has implemented is geofencing. Since Super Cruise is designed primarily as an advanced highway driving assist system for use on limited access roadways, GM is not relying on customers to understand where it does and does not function. Instead, the system will check the navigation map—if the vehicle isn’t on a suitable road, the driver will not be able to activate it. In contrast, Tesla’s operating instructions state that AutoPilot should only be used on divided, limited access roads, but there is nothing in the system to actively prevent a driver from using the system in an urban area or any other roadway that it’s not designed for.

Similarly, Tesla doesn’t really take measures to prevent operators from taking their attention away from the road. Countless videos have been posted by Tesla drivers as they take a nap, read, or even climb in the back seat while using AutoPilot. The research conducted by Bryan Reimer and the Advanced Vehicle Technology Consortium at the Massachusetts Institute of Technology reinforces the idea that even informed drivers will get distracted while using systems like AutoPilot or Volvo’s Pilot Assist.

Improving Safety

Cadillac is installing an active driver monitoring system in the CT6, which will include more prominent alerts if the operator does not remain engaged while using Super Cruise. If the driver does not respond, the car will pull to the side of the road and come to a safe stop.

GM safety engineers have also addressed the issue of the inevitable mechanical failure. When fully autonomous vehicles arrive, they will require systems that can maintain control during a failure mode until the vehicle is safely stopped. One of the key safety failure modes for a system like Super Cruise is the electrically assisted steering.

One of the optional features on the currently available CT6 without Super Cruise is the Active Chassis Package, which includes a rear-wheel steering system to aid low-speed maneuverability and high-speed stability. This rear steering system will be included on the CT6 with Super Cruise. While the rear steering is not designed to provide the same full maneuvering capability of the normal front steering, it will be sufficient to safely steer the car to the side of the road in the event of a front steering failure.

We won’t have an opportunity to fully evaluate the capabilities of Super Cruise until sometime next year, but it does inspire some confidence that GM is at least thinking about and trying to address both human and mechanical failure modes before putting the system into customer hands.

When was the last time you ever actually read an end-user license agreement or terms of service before clicking “Accept” to install a piece of software or join the latest social network? Odds are that unless you are a lawyer, the answer is never. The technology companies that make these products would probably like it to stay that way. However, in the world of the self-driving car, that is not an acceptable policy. The tragic death of a Tesla Model S driver in Florida highlights the need for all automakers to be more open and transparent about the limitations of autonomous technology.

Revolutionary (When It Works)

It seems that barely a day goes by when we don’t get a breathless press release from an automaker, supplier, technology company, or Silicon Valley startup about the amazing progress that they are making on self-driving technology. You can already go out today and purchase vehicles from a number of brands that promise at least partial autonomous capability, and full autonomy is being targeted by the end of this decade. While Tesla Autopilot, Volvo Pilot Assist, and other similar systems seem truly magical when they work as advertised, there are far more scenarios where these systems do not function at all.

Unfortunately, we have not seen Tesla CEO Elon Musk stand on a stage and tell people not to use Autopilot in the city, on curving rural roads, or in the snow. GM CEO Mary Barra stood on the stage at the 2014 ITS World Congress in Detroit and promised a Cadillac with hands-off Super Cruise capability in 2016. I’ve experienced prototype systems from Toyota and Honda and driven production systems from Tesla and Volvo, and when they work, they are incredibly impressive.

I am an engineer by training and technology analyst by trade, and I have a much greater understanding than the average consumer about how these systems work. As a result, I can never truly relax with these systems because I’m always on the lookout for the failure mode, and they are numerous. Unless very explicitly told, the average consumer will be so excited by the prospect of turning over control to a computer that they will not pay any attention to the warnings that Autopilot is very much in beta before enabling it. Volvo doesn’t even give that warning before allowing Pilot Assist to be engaged.

Mainstream Customers

Tesla is fortunate that many of its existing customers are early adopters that expect technology to be imperfect, although most of them probably don’t expect to be at risk of injury when it fails. When the Model 3 arrives and mainstream consumers try Autopilot and find its limitations, they aren’t likely to be as forgiving, and the same is true for every other automaker offering autonomous features. Navigant Research’s Autonomous Vehicles report projects more than 4 million autonomous-capable vehicles to be sold by 2025. Those customers need to know what the systems can do—and, more importantly, what they cannot.

We don’t yet know all the details of what happened in the tragic crash in Florida. Similar accidents where one vehicle crosses a highway divider happen all the time, and fatalities occur when humans are in control. What we do know is that we are far from a time when we can just sit back and relax and let the computer do the driving. Every company involved in this space needs to be far more upfront with consumers about this technology can do or risk poisoning the market.

For many years after J.D. Power and Associates began conducting its Initial Quality Study (IQS) 3 decades ago, most problems reported by customers in the first 90 days of vehicle ownership were either defects or non-functional features. However, in the past decade, the nature of reported problems has shifted toward what J.D. Power calls design-related issues. This could pose a serious problem for manufacturers as they rush to introduce autonomous driving technology.

At a recent meeting of the Automotive Press Association in Detroit, J.D. Power vice president Renee Stephens presented the 2016 IQS results. The industry as a whole improved by 6% in 2016 to just 105 problems per 100 vehicles, the best improvement in 7 years. Among the reported problems, those that fall into the audio, connectivity, electronics, and navigation areas continue to represent the largest category of complaints.

Voice recognition and connected devices still befuddle consumers. Numerous manufacturers including Ford have seen ratings decline in past years as a result of difficulties using infotainment systems. “Expected reliability remains the most important consideration when purchasing a new vehicle, cited by 49% of owners,” said Stephens. “It’s critical that technology be implemented correctly or consumers lose trust.”

Potential Problems

An increasing number of new vehicles now include advanced driver assist systems (ADAS) such as adaptive cruise control and lane keeping aids. However, if features don’t work as expected by the consumer, they often get turned off after a few false positives or surprises. This highlights a potentially serious problem for the auto industry in the coming decade as semi and fully autonomous systems are increasingly rolled out in the marketplace. Navigant Research’s Autonomous Vehiclesreport forecasts that nearly 5 million autonomous vehicles are expected to be sold in 2025, a volume that is expected to grow to more than 40 million in 2030.

Regardless of current ADAS and whether future autonomous systems work as the engineers intend them to, it is absolutely imperative that they work as consumers expect. Autonomous capability will add significant cost to vehicles, and until there is a shift toward on-demand mobility services, consumers will have to absorb that cost. If their experience with the stepping stone technologies is excessively negative, the market will reject these technologies.

Contradictory Views

This will be particularly true if consumers realize that autonomous systems don’t work at all in the scenarios where they are most likely to want to hand over control, such as in poor weather. A major market force for automated driving is improving safety. Related to the general functionality of these systems is the problem of ethics where, as is often the case, the public has contradictory views. A new study by MIT professor Iyad Rahwan shows consumers want autonomous vehicles to minimize casualties in the event of unavoidable crashes. However, that only applies if that person is not the potential casualty. It comes down to protect everyone—but protect me first.

If society as a whole is ever going to benefit from the potential of autonomous vehicles in reducing collisions, congestion, and energy use, much will have to change in society. Consumers will have to be educated in how these systems work so that expectations can be set appropriately. If the bar is not adjusted, consumer complaints in IQS and other studies will skyrocket, and this technology could die on the vine.