Published Paper Details:

We developed a convolutional neural network (CNN) to convert front-facing camera data straight from raw pixels to steering commands. This end-to-end strategy turned out to be unexpectedly effective. The machine learns to drive in traffic on local roads with or without lane markings, on motorways, and with the least amount of training data from people. By using simply the human steering angle as the training signal, the system automatically picks up internal representations of the necessary processing stages, such as spotting useful road elements. For instance, we never expressly trained it to recognize the outline of a road. Our end-to-end approach simultaneously optimizes all processing steps in contrast to explicit decomposition of the problem, such as lane marking detection, path planning, and control. We contend that doing so will eventually result in bigger systems and better performance. The internal components will self-optimize to maximize system performance rather than improving intermediate criteria chosen by humans, such as lane detection, leading to better performance. It makes sense that these criteria be chosen for simplicity of human interpretation, but this does not imply that they will always result in the best system performance. Because the system learns to solve the problem with the fewest amount of processing steps possible, smaller networks are made possible. For training, we used an NVIDIA Dev Box and Torch 7, and for navigation, we used an NVIDIA DRIVETM PX self-driving car computer running Torch 7.