Tesla is transforming the challenge of autonomous driving into a purely AI-driven problem using an "end-to-end" neural network, rather than an engineering issue requiring countless engineers to write rules. They believe that the industry's mainstream approach of modularizing perception, prediction, and planning not only makes the system cumbersome but also imposes clear limitations. End-to-end AI models, they argue, are the correct solution for autonomous driving.
On October 25, Ashok Elluswamy, Tesla's AI lead, published a lengthy post on X, revealing the technical methodology behind Tesla's Full Self-Driving (FSD) system. The post was highly informative.
In simple terms, Tesla is transforming autonomous driving into a pure AI problem using an "end-to-end" neural network, rather than an engineering problem requiring countless engineers to write rules.
They argue that the industry-standard modular approach, which separates perception, prediction, and planning, is not only cumbersome but also has clear limitations. In contrast, end-to-end AI models are the right solution for autonomous driving.
The key points of the article are as follows:
Core debate on technical approaches: Why must it be "end-to-end"?
The mainstream industry approach involves a three-part solution: perception, prediction, and planning, with each module operating independently before being integrated. Tesla believes this method creates complex interfaces and is difficult to optimize. On the other hand, an "end-to-end" AI model directly "sees" pixels and "outputs" driving commands in one step, allowing the entire system to be optimized holistically. This is not only about solving the driving problem but also about positioning Tesla on the right side of scalability amidst AI’s “bitter lessons.”How does AI address the challenge of 'humanity'?
Real-world driving is full of "mini trolley problems," such as whether to drive through a puddle or veer into the opposite lane. These subtle decisions, which require weighing pros and cons, are difficult to hard-code. However, by learning from vast amounts of human driving data, AI can implicitly learn driving strategies aligned with human values.AI can read between the lines.
FSD can distinguish between "a group of chickens wanting to cross the road" and "a group of geese just loitering by the roadside," thereby making different decisions. This understanding of 'potential intent' is difficult to convey in a modular system but can be effortlessly understood and processed within the 'latent space' of an end-to-end model.Data deluge overwhelms the 'curse of dimensionality'
FSD processes up to 2 billion inputs per second from cameras, maps, audio, and other sources, instantly compressing them into two commands (steering and acceleration). The sole solution to this challenge is Tesla's fleet producing a 'data Niagara Falls' equivalent to 500 years of driving time daily, intelligently filtered to feed the AI with the most valuable data.The effect of 'brute force achieving miracles': Anticipating your anticipation
Vast amounts of data have trained remarkable generalization capabilities. In one slippery rainy-day case, FSD began decelerating 5 seconds before an accident occurred, as it anticipated that the vehicle ahead would lose control, 'bounce off the wall,' and return to its lane. Such advanced anticipation of 'second-order effects' is unattainable for traditional solutions.Two keys to unlocking the AI 'black box'
To address the challenges of debugging and interpreting end-to-end models, Tesla has configured the model to output not only driving instructions but also comprehensible 'intermediate results.' Two approaches were highlighted in the text:
Visual reconstruction: Utilizing 'generative Gaussian splatting' technology, a dynamic 3D model of the surrounding environment is generated in real-time from camera footage within 220 milliseconds, allowing engineers to 'see' the world through AI's eyes.
Language explanation: Training AI to explain its behavior using natural language. A compact language reasoning model has already been operational in FSD version 14.x.
The hardest challenge lies in 'evaluation'
No matter how high the model scores, it is useless unless it performs well in real-world evaluations. To this end, Tesla has developed a 'Neural World Simulator.' This simulator itself is also a powerful AI capable of generating hyper-realistic virtual worlds in real time. It can not only reproduce historical data but also create various extreme accident scenarios to conduct 'hell mode' stress tests on FSD (Full Self-Driving). Essentially, Tesla has created a surreal 'driving game' for FSD, allowing it to continuously 'train and upgrade' 24/7.The ultimate goal of this technology stack: one AI system to handle everything.
This methodology is not only applicable to automobiles but can also be seamlessly transferred to Tesla's 'Optimus' humanoid robot. The article demonstrates that the simulator can already generate scenes of Optimus navigating within a factory, proving the versatility of the technology, with the ultimate aim of solving the problem of artificial general intelligence (AGI) in the real world.
Note: Ashok Elluswamy is Tesla’s Vice President of AI Software, overseeing Tesla’s artificial intelligence operations. He was promoted to his current position in 2024, having previously served as Director of Autonomous Driving Software. Since 2022, he has reported directly to Elon Musk and participated in the early development of Tesla’s autonomous driving system.
Original text follows:
[Tesla's Approach to Autonomous Driving]
This week, I had the privilege of representing the @Tesla_AI team at the International Conference on Computer Vision (ICCV), where we showcased some of our recent work. In this condensed version of the presentation, we will explore key aspects of Tesla’s approach to solving the autonomous driving challenge.
As many are aware, Tesla employs an end-to-end neural network to achieve autonomous driving. This end-to-end neural network takes in pixel information from multiple cameras, vehicle kinematic signals such as speed, audio, maps, and navigation data, and ultimately outputs control commands to drive the car.
Why end-to-end?
Although Tesla firmly believes in the end-to-end neural network approach, this is by no means an industry consensus for achieving autonomous driving. Most other entities developing autonomous driving systems adopt a method that relies on a large number of sensors and a modular approach. While such systems may be easier to develop and debug initially, they inherently possess numerous complexities. Compared to this baseline, the end-to-end approach offers several advantages, including but not limited to:
Encoding human values is extremely difficult. Learning these values from data is much easier.
The interfaces between perception, prediction, and planning are poorly defined. In an end-to-end network, gradients can flow all the way from the control end to the sensor input end, enabling holistic optimization of the entire network.
It can be easily scaled to address fat-tail and long-tail problems in real-world robotics.
Homogeneous computation with deterministic latency.
Overall, in terms of scalability, this adheres to the correct guidance of the 'bitter lesson.'
The following examples illustrate this point.
Example 1:
In the example below, the AI must decide whether to drive through a large puddle or cross into the oncoming lane. Typically, crossing into the oncoming lane would be highly undesirable and potentially dangerous. However, in this case, the vehicle has sufficient visibility to determine that no oncoming traffic is expected in the foreseeable future. Additionally, the puddle is quite large and best avoided. Such trade-offs are difficult to articulate using traditional programming logic, yet they are relatively straightforward for a human observing the scene.
The classic 'trolley problem' is often considered a rare issue that autonomous vehicles would seldom encounter. However, the opposite is true. Autonomous vehicles constantly face 'micro-trolley problems' as illustrated above. By training on human data, robots can learn values that align with human principles.
Example 2:
It is difficult to establish a clear interface between the two modular units of 'perception' and 'planning.' In the following two scenarios, a flock of chickens wants to cross the road in one scene, while a group of geese simply wants to stay in place in another. Creating an ontology for these modular units is quite challenging. This 'flexible intent' is best transmitted in an end-to-end, latent manner.
For all these reasons and more, Tesla has adopted an end-to-end architecture for autonomous driving. That said, there are still many challenges to overcome in building such a system. We will discuss a few of these challenges next.
1. Curse of dimensionality
To operate safely in the real world, it is necessary to process inputs at high frame rates, high resolutions, and with long contextual histories. If we make a reasonable assumption about the size of 'input tokens,' such as a 5x5 pixel image patch, we would end up with the following number of tokens:
7 cameras x 36 FPS x 5 million pixels x 30 seconds of historical data / (5x5 pixel image patch)
Navigation maps and routes for the next few miles
Kinematic data at 100 hertz, such as velocity, Inertial Measurement Unit (IMU), and odometry.
Audio data at 48 kilohertz.
This is equivalent to approximately 2 billion input tokens. The neural network needs to learn the correct causal mapping to reduce these 2 billion tokens into two tokens—namely, the vehicle’s next steering and acceleration commands. Learning the correct causality without picking up spurious correlations is an extremely challenging problem.
Fortunately, Tesla, with its vast fleet, has a Niagara Falls-like torrent of data. The entire fleet collectively generates mileage data equivalent to 500 years of driving every day. Not all data is valuable, nor can all of it be collected. Therefore, Tesla employs a sophisticated data engine pipeline to filter the most interesting, diverse, and high-quality data samples. Below are examples of such data.
If you train with such data, you achieve strong generalization capabilities for corner cases, which are difficult to attain using other methods. Here is an example showing how the AI model learns to proactively avoid a potential collision. What is impressive in the video is that the AI reacts around the 5th second, long before it becomes clear that the situation could escalate into a collision. The AI needs to understand: it is drizzling, the vehicle ahead may be losing control and skidding, it might hit the guardrail and bounce back into the path of the self-driving vehicle, so it should cautiously apply the brakes now. Only an exceptionally capable AI system can predict such second-order effects this far in advance.
2. Interpretability and Safety Guarantees
Debugging such an end-to-end system when vehicle behavior deviates from expectations can be difficult. However, in practice, this is not a major issue because the model can also produce interpretable intermediate tokens. Depending on the context, these intermediate tokens can also serve as reasoning tokens.
One of the tasks is Tesla's "Generative Gaussian Splatting." Although 3D Gaussian splatting technology has made significant progress in the field of computer vision in recent years, it relies on large baseline camera perspectives to achieve good performance. Unfortunately, typical vehicle motion trajectories are quite linear, and running traditional Gaussian splatting can lead to poor reconstruction quality, especially from novel viewpoints. These 3D Gaussian splats also require good initialization from other pipelines, and the total optimization time can last up to tens of minutes.
In contrast, Tesla's generative Gaussian splatting demonstrates excellent generalization capabilities, with a runtime of approximately 220 milliseconds. It does not require initialization, can model dynamic objects, and can be jointly trained with end-to-end AI models. Notably, all these Gaussians are generated based on cameras configured in mass-produced vehicles.
In addition to 3D geometry, the system can also incorporate video grounding to perform reasoning using natural language. A smaller version of the reasoning model is already operational in FSD v14.x.
3. Evaluation
The final and most challenging task is evaluation. Even with high-quality datasets, the loss from open-loop predictions may not correlate with superior real-world performance. The evaluation process needs to be diverse and mode-covering to enable rapid development iterations. This work is labor-intensive and requires significant effort to achieve a high signal-to-noise ratio in evaluation metrics.
For this reason, at Tesla, we have developed a neural world simulator. This simulator is trained using the same massive dataset that we have curated. However, instead of predicting actions for a given state, it synthesizes future states based on the current state and the next action. This can then be connected to an agent or policy AI model to run in a closed-loop fashion, thereby evaluating performance.
This world simulator is entirely trained by Tesla and is used to generate all camera and sensor data for vehicles. It is causal and responsive to the commands of driving policy models. It operates at high speed while synthesizing high-resolution, high-frame-rate, and high-quality sensor data.
Here is an example of a one-minute rollout of this neural simulator model.
[Video content description: A fully AI-generated simulated driving video lasting just over a minute. The top shows the front camera view, the middle shows side views, and the bottom shows rear views. The visuals are realistic, with lighting, vehicle dynamics, and environmental details highly simulating the real world.]
This simulation can be used to validate new driving models against historical data.
[Video content description: Starting from the same real video clip (marked with a green square), the simulator generates two completely different but physically logical future trajectories based on a series of varying actions output by the new model.]
Additionally, we can artificially synthesize new adversarial scenarios to test additional edge cases.
[Video content description: Starting from the same initial video, a background vehicle in the simulator is set to act adversarially (e.g., suddenly cutting in) to test the response capability of the FSD model.]
By adjusting the amount of computational resources used during testing (test-time compute), the same model can simulate the world in real time. Below is an example where a person is able to drive for over six minutes, with all eight camera views (24 frames per second) fully synthesized in real time by a neural network. Notably, even during such an extended generation period, the level of detail remains highly realistic.
[Video content description: A demonstration resembling a driving game.]
The most remarkable aspect of all the points above is that they not only address the autonomous driving challenges for vehicles but also seamlessly transfer to Tesla's humanoid robot—Optimus. Here is an example of such a transfer.
The same video generation model is also applicable to Optimus robots navigating within Tesla’s Gigafactories.
[Video content description 1: Shows an Optimus robot walking and performing tasks in a virtual Tesla factory environment within a neural world simulator.]
[Video content description 2: Different movements of Optimus are accurately reflected in the world simulator, demonstrating the simulator’s precise response to robotic behavior.]
Clearly, all the video generation technologies described above are not limited to evaluation purposes. They can be utilized for large-scale closed-loop reinforcement learning to achieve superhuman performance.
Editor/Jeffy
