How do self-driving cars work?

Tesla a car company based entirely on two concepts, electric and self-driving cars. Soon after Elon Musk started the company many automobile giants such as Mercedes, Nissan, Ford etc., started announcing their autonomous car projects. Tech industry titans such as Apple and Google also not wanting to be left out announced to produce fully automatic cars by the year 2020.

I was recently asked the question by someone who was not too knowledgeable about programming but had some idea, "How do self-driving cars work?". 

I had the basic idea that they use semi-supervised machine learning, the program is fed with lots of data about how humans drive cars and also the rules of driving and simulations and test runs are used for it to gather data on its own and learn from itself. But it got me curious and I started my research on how exactly the algorithms work. Here's what I came up with.

Self-driving cars have five core components: Computer Vision, Sensor Fusion, Localization, Path Planning, Control.

Computer Vision is how we use cameras to see the road. Humans demonstrate the power of vision by handling a car with basically just two eyes and a brain. For a self-driving car, we can use camera images to find lane lines or track other vehicles on the road.

An Artificial Neural Network is developed using Deep Learning to distinguish edges and color gradient for the car to identify the lane. It is also used to mark cars on the road surrounding you.

Sensor fusion is how we integrate data from other sensors, like radar and lasers together with camera data to build a comprehensive understanding of the vehicle’s environment. As good as cameras are, there are certain measurements like distance or velocity at which other sensors excel, and other sensors can work better in adverse weather, too. By combining all of our sensor data, we get a richer understanding of the world.

Uber's self-driving prototypes use 64 laser beams, along with other sensors, to construct their internal map; Google's prototypes have, at various stages, used lasers, radar, high-powered cameras, and sonar.

Localization is how we figure out where we are in the world, which is the next step after we understand what the world looks like. We all have cell phones with GPS so it might seem like we know where we are all the time already. But in fact, GPS is only accurate to within about 1–2 meters. Think about how big 1–2 meters is! If a car were wrong by 1–2 meters, it could be off on the sidewalk hitting things. So we have much more sophisticated mathematical algorithms that help the vehicle localize itself to within 1–2 centimeters.

An algorithm which determines the exact position using the sensors by taking into accounts landmarks such as light posts, buildings etc., is used for this purpose.

Path planning is the next step, we chart a trajectory through the world to get where we want to go. First, we predict what the other vehicles around us will do. Then we decide which maneuver we want to take in response to those vehicles. Finally, we build a trajectory, or path, to execute that maneuver safely and comfortably.

Control is the final step. Once we have the trajectory from our path planning algorithm, the vehicle needs to turn the steering wheel and hit the throttle or the brake, in order to follow that trajectory. If you’ve ever tried to execute a hard turn at a high speed, you know this can get tricky! Sometimes you have an idea of the path you want the car to follow, but actually getting the car to follow that path requires effort. Race car drivers are phenomenal at this, and computers are getting pretty good at it, too!

The technology for self-driving cars is not yet at a place where we can rely on it by giving it total control. But it will come around, and when it does I will be eager to see how it changes the world.