If you follow me on Twitter, you may have noticed that I’ve taken a keen interest in the topic of autonomous driving, as well as Tesla’s Autopilot technology and the company more generally. Twitter is rarely an ideal medium for conversation of any length, depth, or nuance – so I thought it might be time for me to revive the ol’ blog and to write about Autopilot. In particular, some things I’ll try to cover include: What is Tesla’s Autopilot? How does it work? Is it safe? How crazy are Elon Musk’s claims about “full self-driving” and “robotaxis”, and what do I think might happen over the next few years?

I wrote most of these entries back in February while on an airplane, but never got around to posting them. I’ll insert a few updates here and there but otherwise this post is pretty much as written at that time.

Disclaimer: I’m an enthusiastic Tesla owner and I own a small chunk of Tesla stock. However, I strive to be objective about the company, and I’ll let you judge my success at that for yourself.

What is Autopilot?

Autopilot is the name Tesla gives to its suite of Advanced Driver-Assistance Systems available today on their vehicles. Rather than having a fixed definition or referring to one specific feature, Autopilot refers generally to the overall hardware + software system, and its capabilities have evolved over time. The capabilities offered by the system today include specific features such as Traffic-Aware Cruise Control, AutoSteer (aka Lane Keep Assist, plus Automatic Lane Changes), Navigate On AutoPilot (automatic navigation-based highway driving), and Smart Summon.

Update: New functionality added since I originally wrote this piece includes “Traffic Light and Stop Sign control” and some recent improvements to speed limit detection.

Traffic-Aware Cruise Control

The most basic capability AutoPilot offers is Traffic-Aware Cruise Control, or TACC. More generally, this is a form of Adaptive Cruise Control. Where regular cruise control simply maintains a set throttle or speed until adjusted or disengaged by the driver, ACC systems measure the speed of a vehicle (or vehicles) ahead and use this to dynamically adjust the cruise control setting.

The earliest implementations of this concept have been around for almost three decades now. The earliest versions used lidar sensors (laser-based range finding) and would only adjust the throttle – they would not apply the brakes. These were very niche offerings, some of which were limited to the Japanese market. In 1999, companies like Cadillac and Mercedes began offering their own systems as high-end options on some of their luxury cars, and their implementations were based solely on a front-facing radar. These gradually evolved over subsequent years, but remained relatively rare in the market. These were squarely positioned as convenience features, primarily designed to remove the need for small throttle adjustments at highway speeds. Most systems couldn’t stop the car, and they were not intended or able to avoid accidents.

Later versions of these systems would gain additional abilities, including the ability for some systems to stop the car, or even to stop and then resume motion in traffic jams. Eventually these systems began to offer forward collision warning and emergency braking capabilities – features which have become fairly ubiquitous in recent years.

Tesla’s TACC mode is similar in concept and implementation. It is, however, one of the more advanced (if not the most advanced) system of this kind available today. It uses both radar and vision to track multiple vehicles, including vehicles ahead of the one you’re following (partly by bouncing radar underneath the car directly in front of you), as well as vehicles in adjacent lanes. Today’s version of the AutoPilot software will even predict “cut-ins” (e.g. a car changing into your lane in front of you), regardless of whether the car signals its lane change. It will also adjust its speed automatically based on other factors including its knowledge of speed limits via its map data, the curvature of the road, and the speed of traffic in adjacent lanes. This doesn’t eliminate all need to adjust the chosen speed manually, but it does cover a lot of the most common cases.

However, as I’ll discuss in more detail later, it is not able to do all of these things perfectly, and it is not able to stop for all obstacles in the road. For example, cut-in detection will sometimes not notice a car merging in as soon as you’d probably like, and if you come around a curve at significant speed, it may not notice some stopped traffic ahead of you until the last second (or in rare cases, at all). Tesla is extremely clear about this in the manual, and in the user interface for the system, and it is important for any driver using this (or any Adaptive Cruise Control system) to remember that they’re ultimately responsible for the operation of their vehicle.

Tesla’s owner’s manuals say: “Traffic-Aware Cruise Control is primarily intended for driving on dry, straight roads such as highways and freeways. It should not be used on city streets.” However, like standard cruise control found in most cars, where and when it is enabled is up to the driver. In fact, Tesla’s cars do not offer a non-traffic-aware cruise control feature – AutoPilot’s TACC is the only cruise control in a Tesla.

AutoSteer

AutoPilot’s next more advanced mode of operation is called AutoSteer. AutoSteer refers to the ability of the system to track the road and lane markings, and to steer the vehicle to stay within them. AutoSteer cannot be enabled without TACC, though the driver can temporarily override speed and acceleration using the accelerator pedal without disengaging AutoSteer.

Several car manufacturers offer various Lane Keep Assist systems in cars you can buy today. The exact functionality and implementations vary a good deal. Many operate only at specific speeds, or where lane markings are pristine. In my experience some barely seem to work at all, to the point where I believe they are at best useless and at worst dangerous. Others are quite competent, and the quirks in their behavior range from unpleasant (e.g. “bouncing” between sides of the lane) to subjective matters of preference (e.g. strict adherence to the center of the lane).

Tesla warns drivers about AutoSteer’s limitations in both in the owner’s manual and in a screen you must accept before enabling the AutoSteer feature in the car’s Settings app. This must be done while parked, and separately for each driver profile. Specifically, they say:

In practice, AutoSteer can be enabled on most roads, so long as AutoPilot can identify the boundaries of the lane you’re in (which it can do surprisingly well even with little in the way of markings). If you are not on a supported highway, however, it will limit the car’s maximum speed while on AutoSteer – I believe to 5MPH over the known speed limit for the road, or 45MPH for places where the speed limit is not known. However, it’s really only useful for highway operation, and in particular it is best suited to divided highways with no cross-traffic (e.g. with on-ramps and off-ramps instead), just as Tesla says. Any use outside of those conditions requires extra vigilance from the driver.

AutoSteer is in my experience quite adept at identifying lane boundaries even with faded or absent lane markings, and will use road edges and other indicators to figure out where it should go. It’s also very impressive at tracking the lane in all manner of weather conditions where I’ve used it. However, it is not perfect. Its behavior when lanes merge or get extra wide can be off-putting (e.g. in some cases it will try to move to the center when really it should stay to one side while the lane shrinks back to normal size).

In rare cases where there are multiple sets of lane lines (e.g. where old sets are visible after new ones have been put down), it can become very confused. Though to be fair, in many of those places it’s easy for human drivers to get confused too. When you hit this situation, it will make a loud alert sound and flash a big warning on the screen telling you to take over immediately. Generally it will still follow the correct path during this time, though the wheel may twitch back and forth a bit until you take control.

While engaged, AutoPilot will continually check to determine if your hands are on the steering wheel. The way it does this is by measuring torque (turning force) on the steering wheel itself, or interaction with any of the steering wheel buttons or scroll wheels. So if your hands are just gently resting on it while driving straight, it can’t tell if they’re there or not. Because of this, the system will periodically “nag” you to apply some pressure to the steering wheel.

The nag behavior is very dynamic, and I think quite clever. If you’re going straight and the system is confident about what’s ahead, it will go 30 seconds or more without detecting any torque before it nags you – the maximum length of time varies depending on speed. However, if the car turns the wheel at all, it will immediately know if your hands are there or not (based on whether there’s any resistance) and will immediately nag you if they are not. Further, it will nag you immediately any time it becomes at all uncertain about the road markings or what lies ahead. In general, I’m able to rest my left hand on the wheel in a way where I do not get nagged very often. When I do, it’s just a matter of applying a small amount of pressure momentarily and the message immediately disappears.

If you fail to respond to the nag message, it will quickly get more aggressive about getting your attention, first flashing the background of the whole “instrument cluster” blue, then making urgent audible noises, up until a point where it will begin to slow the car to a stop and activate the hazard lights. I’ve never let it get that far, but I’ve seen videos showing what happens if you don’t respond. This seems sensible, and I think it is actually something that could be valuable in all cars, regardless of AutoSteer type functionality. Fatigued drivers all too often fall asleep at the wheel with no automation or with regular cruise control enabled, or could be suddenly incapacitated by an urgent health issue. Having any car able to safely come to a stop in this situations can save lives.

AutoSteer also supports Automatic Lane Changes. While AutoSteer is enabled, you can simply activate the turn signal in the direction you’d like to change lanes, and the AutoPilot system will execute the change. If the lane is clear, it will happen quite immediately. If it is not, the driver visualization screen will show the lane boundary and any cars in the way in red. AutoPilot will attempt to adjust speed to find an opening, and move over once it finds a spot. In some cases, it can be helpful to apply the accelerator to encourage AP to move in front of a car rather than slowing down and trying to move over behind it. In these situations, AutoPilot does what I think is an impressive job of melding “(hu)man and machine” operation, and what could be a complicated UX nightmare actually seems to be handled very seamlessly in my experience.

Navigate On AutoPilot

The most advanced mode of operation AutoPilot offers today is called “Navigate on AutoPilot” (or “NOA”). This is a pretty seamless addition on top of AutoSteer while a navigation route has been selected. When enabled, and on supported highways in supported conditions, AutoPilot will automatically take exits and interchanges to follow your chosen navigation route. It will also recommend or even automatically execute (if enabled in a settings page) lane changes to overtake other cars, move back out of the passing lane, and to follow the route (i.e. to be in the correct lane for an upcoming exit).

NOA is really good at some things and pretty good at others, with some significant caveats. Its performance and usefulness is very dependent on having accurate map data and GPS signal. It also will deactivate itself or temporarily limit its functionality if it detects certain conditions – including particularly poor weather, or some construction zones. In these cases, it will deactivate the Navigate On AutoPilot functionality, but revert to the normal AutoSteer behavior. These situations are communicated to the driver via recognizable sounds and a message about what happened, but there is no urgent alert or panic from the system as it will continue driving in the current lane, subject to normal AutoSteer behaviors and limitations. The system will also automatically resume NOA functions if/when the condition passes.

In the default configuration, NOA will execute maneuvers to take off-ramps without any kind of intervention, but it will only execute lane changes with a confirmation from the driver. It will make a sound, vibrate the steering wheel, and show an indicator that a lane change is suggested, and the driver can confirm it either by pressing the turn signal stalk in the corresponding direction, or by pressing the same button or stalk that engages AutoSteer (varies by model). It will then activate the turn signal, execute the lane change (same as if you used the Automatic Lane Change feature in regular AutoSteer mode), and then deactivate the turn signal.

If, however, you change the advanced setting to disable lane change confirmations (as I have), the car will behave differently. When NOA wants to make a lane change, it will make a sound, vibrate the steering wheel, and display a message on the screen just like in the default configuration – but it will then also turn on the corresponding turn signal. At this time it will check for torque on the steering wheel indicating that your hands are there and (roughly) that you’re paying attention. If no torque is detected, it will jump to the second level of “nag” mode immediately, and it will not change lanes until your hands are detected. Even if pressure on the steering wheel is detected right away, it will wait a minimum amount of time (5 or 6 seconds?) before beginning the lane change – what feels like an eternity compared to the usual Automatic Lane Changes initiated by the driver. Otherwise, the behavior is the same. Some consider the need to apply torque on the wheel to be a sort of “confirmation” despite Tesla calling this a “no confirmation” mode, and I think they’re mostly right in practice, but Tesla is right technically. One key distinction is that if you have your hand resting with enough weight on the wheel to be detected, it doesn’t matter which direction that weight is applying torque on the wheel – it can be the opposite direction of the lane change. It is, however, pretty clear that the system is being extremely cautious and wanting to give you ample time to disagree with what it’s about to do and stop it before it starts.

In some ways, that makes sense and is probably necessary for the current version of the software. On the other hand, it’s kind of disappointing and actually makes this feature less valuable. The delay in executing the lane change is often enough that the gap it could’ve moved into is gone – particularly around Seattle where drivers see turn signals as a “please come block me” message. Other times it works out fine but just makes you look like a timid driver. In the worst case, someone may think they’ve given you enough time, and decide that you must not have wanted to change lanes after all. Then you start moving just as they decided to pass, so then AP aborts the maneuver, and basically everybody gets annoyed.

In my experience, NOA is very good at taking exits and interchanges, and generally pretty good at overtaking cars. The main problems I have with it fall into the following buckets:

1. Relatively minor annoyances like wanting to get over to the rightmost lane unnecessarily early before an exit. Most often I’d prefer it to wait until after we’ve passed the last on-ramp where cars are merging in before deciding to move over for an exit, assuming I’m in a middle lane or passing slower traffic. It’s also pretty inconsistent about this – sometimes wanting to get over miles ahead of an exit, and sometimes not.
2. Unexpectedly aborted lane changes. These are cases where the car starts executing a lane change in its normal fashion but then panics and abruptly swerves back to the center of the original lane. In theory this shouldn’t be a safety concern as it hasn’t fully left the original lane and any driver has the right to abort a lane change for various reasons (and this happens from time to time). But the abrupt manner in which it does it is off-putting, and likely disconcerting to other drivers. This used to be particularly problematic for me on the 520 bridge for some reason, and I suspect due to readings from the ultrasonic sensors picking up either the guard rails or the raised lane markers on the ground. I haven’t hit this lately or at least not regularly, so it may be something they’ve largely fixed. But that’s hard for me to say with any certainty as changes to the main trouble spots on my commute and to my usage patterns may be partly responsible.
   Update: It’s been about 7 months since I originally wrote that down. Since then, I haven’t observed a single case of this behavior. A couple of times I’ve now seen the system hesitate for an instant halfway through a lane change, but then continue successfully rather than jerking back to the original lane. It seems like they may have fixed this issue or at least made it a lot less common or jarring.
3. Out-of-date maps. Due to construction on the west side of the 520 floating bridge, the navigation data used for routing and NOA functionality no longer matches reality. To its credit, I haven’t seen the car do or try to do anything unsafe due to this, however it does result in the navigation system suddenly deciding I’m on a different road (not the highway) and rerouting me, and this definitely confuses the NOA system. It also triggers AutoPilot to reduce its speed because it suddenly thinks it’s on an off-ramp (accidentally matching the posted temporary speed limit due to ongoing construction, but not at all matching the speeds everyone drives there). Because of this, I now disable NOA during this section of my drive, and resume it afterward.
4. Failures to take exits, where you miss your exit because NOA got confused and passed it. This has been extremely rare but I think it’s happened once or twice. Just as often, I have thought it was going the wrong way and made a last-minute change myself, only to realize I was mistaken and NOA was correct. So I guess we’re about even on that count.
5. The really confused stuff. I can only think of one example, but I did once activate NOA very close to my exit for work in some heavy rain, actually unsure if it would even enable or give me the “poor weather” message. However, it did enable, but then proceeded to get confused and apparently think that I was driving on the shoulder or a non-existent lane that was ending, and not the rightmost lane where I was and needed to be. It then wanted very badly to change lanes to the left. I of course could override it and did so, but it was one of the more disconcerting experiences I’ve had.

Smart Summon

Smart Summon is really the closest thing to autonomous driving offered by Tesla today, as it enables the car to drive without a driver being present in the vehicle. Technically speaking though, Tesla considers you to be operating the vehicle remotely – and responsibility still falls on you the “driver” operating the vehicle remotely via the app. This can make it pretty terrifying, and is why I suspect usage of it is pretty limited.

The basic idea is that you launch the smartphone app, select Summon -> Smart Summon, and then you either use the “come to me” button or you place a pin on a satellite view map of where you are. The app requires you be within a certain range of the car, and that the car is not on a public road. It is intended only for use in parking lots or private driveways.

The app will show you a line depicting the planned path the vehicle will try to take, and sometimes you can futz around with the pin to adjust its plan before it starts. Its actual path will vary depending on what it encounters along the way, but it’s good to look at the planned path and make sure it is sane before you start.

When you’re ready, you press down on the “summon” button, and hold your finger down. The car will light up and if the coast is clear, it will quickly begin moving (usually backing out of its parking spot). I think parking head-in is preferable if you’re going to use Smart Summon, as the car has a better view of the ground behind it – there’s a blind spot just in front of the bumper in the forward direction, where it only has ultrasonic sensors to detect potential obstacles. When backed into a spot, I believe the car will actually try to reverse a little bit before going forward (if it has clearance behind), to give itself a better view of what’s immediately in front of it before proceeding.

The car will then leave the parking spot and make its way to you or the pin you placed on the map. In general it will use its turn signal when making turns (even when a human driver probably wouldn’t), and will stop at “intersections” to make sure they’re clear before proceeding. It seems to try to stay to the right of the road/aisle, but it doesn’t always get this quite right. It will sometimes stop or slow down for no apparent reason, and will sometimes stop more abruptly than necessary when it sees another car move near its path. It will also sometimes wait longer than necessary once the path is clear. In general, these are cases where it’s overly cautious, and frankly I’m okay with that at this point.

I’ve tried the feature out many times, and in a few cases it’s been useful (e.g. kept me dry during some rain), but mostly it’s just neat. I’d estimate my success rate with it as:

~50% of the time it basically exactly what I want
~30% of the time I get scared and give up, despite it not doing anything wrong
~20% of the time it gets confused and starts going the wrong way, at which point I stop it and make an embarrassing dash to “rescue” it and not be in anybody’s way

Most of my usage has been in Microsoft parking lots (before we started working from home in February), usually in the evening when there aren’t many pedestrians or cars around. For that 30% of time where I give up, it’s usually because my vantage point makes it hard to tell how close it is getting to a parked car or curb, or I see people or fast-moving cars coming toward it and I get too nervous. I’ve never seen it take any action that would result in injury or damage, but it’s definitely terrifying to have your (very expensive) car moving on its own even remotely near to a wall or another car.

For the most part, my concerns while using it are that I’ll annoy or inconvenience others. Fortunately, the few times where I thought someone might be annoyed or alarmed, they actually just wanted to come tell me how amazed they were about what they’d just seen. I suspect some of this is a matter of where I’ve been while using it, though.

As with other AutoPilot features, I expect this to improve over time and hope that eventually it will become something that can be more easily relied upon and less of an anxiety-inducing party trick.

Traffic Light and Stop Sign Control

The most recent addition to the AutoPilot suite is called “Traffic Light and Stop Sign Control”. This is the latest “beta within a beta” feature to be added for Tesla owners who have purchased the most advanced “Full Self Driving” package (more on that name coming in another post) with the latest AutoPilot hardware.

When enabled, this feature allows the car to stop itself for traffic lights and stop signs in both the TACC and AutoSteer modes. When first released, this feature meant the car would stop at all traffic lights, regardless of what color/state they were in. So if you approached a green light, the car would tell you it’s about to start slowing down, and then you’d have to either hit press the stalk or briefly tap the accelerator to tell it you wanted it to proceed. If you did this and then the light changed yellow or red, it would again begin to stop. When it detected a light as red, it would not proceed unless you manually took over accelerating.

In an update a month or so later, this functionality was updated such that the car will now proceed through a green light automatically if it sees another car in your lane go through it. Otherwise the behavior is as before. I think this approach makes a lot of sense, and is a great example of starting with a conservative and safety-biased behavior which enables them to measure how often the driver agrees with the system, then iterate until their safety goals are met, before enabling more complete automation of cases where higher risk is inherently present.

While it may sound like this requires a lot of “baby sitting”, I’ve found it really works quite well. Most of my usage of this has been in the TACC mode, as AutoSteer isn’t currently useful or appropriate for the kinds of roads where this is useful. It does need more work, and I’d only recommend it to “advanced” AutoPilot users who want to try the latest and greatest but who will be very actively engaged with the system, and take the necessary care in its operation (plus being willing to put up with its foibles).

As with AutoPilot in general, this is not a “set it and forget it” system. Use of this, even with just TACC, means engaging and disengaging it as needed throughout your drive. When it stops at stop signs, for example, you often will need to disable it so that you can inch forward a bit to see what’s coming before you proceed. It doesn’t (yet) do anything like that on its own. It stops at the stop line, and when you tell it to go it will go. It’s also not really ideal for turns, even though it does generally maintain a lower speed and/or match the speed of a car turning in front of you – it just isn’t very natural yet and works best if you disable it, take the turn yourself, and then re-enable it.

However, on long county highways and country roads, like the one we took to our campsite for Labor Day weekend, it makes for a nice upgrade over “plain” cruise control or even “classic” TACC. You can go for very long, straight stretches without having to touch the accelerator. Just minding the steering and paying attention in case you need to brake for something unexpected.

It will be interesting to see what Tesla does next with this. They say they’re working on supporting turns, which seems like it will take things to a new level if they can make it work well. This is clearly a very complicated thing to do which they need to get absolutely right in order for it to be used safely. I wouldn’t be surprised if, just like with Traffic Light & Stop Sign Control, they start with the simplest case. Requiring the driver to confirm that it’s okay to proceed, and perhaps only supporting certain intersections, right turns only, or some other conservative limitation like that. Then over time they can get progressively more sophisticated.

More to come

I’ve already written up a great deal about the topic of safety, including some analysis of the data that’s been made available by Tesla and others. I’ll do a pass through that soon and get it posted here as Part 2.