Lima, Peru — Autonomous driving has been an ambitious promise made by some of the world’s leading technology firms and startups, including giants like Google, Tesla and Uber, however, its mass adoption remains out of reach.
Since 2022, many autonomous vehicle (AV) companies have shut down, and major names like Google’s Waymo, Elon Musk’s Tesla, and General Motors’ Cruise, despite receiving billions of dollars in funding, are downsizing their teams and struggling to maintain the vision of the self-driving car.
Despite significant progress in computer vision, such as the development of Detectron2, accidents involving AVs continue to occur. In October of last year, Cruise recalled its 950 autonomous models to update the software after car dragged a woman down a street in San Francisco.
What causes these failures, and why can’t machines replicate human behavior behind the wheel?
Arturo Deza, a former postdoctoral researcher at the Massachusetts Institute of Technology and a professor at Lima’s University of Engineering and Technology (UTEC), is on a mission to find out.
His startup, Artificio, is developing benchmarking platform specifically for autonomous driving in challenging conditions. The company is headquartered in Lima, which navigation app TomTom recently named as the city with the worst traffic in Latin America.
The challenge of autonomous driving
According to Deza, autonomous driving “has failed due to an overpromise that artificial vision systems were already solved, as if computer vision were fully developed.”
He told Peru Reports that the promise that a computer can identify common objects is a stretch, and there “are still conditions where the stimulus is ambiguous.”
While computers can identify most objects well, he explained, they often fail in complicated situations, such as driving in congested traffic.
“Many autonomous car projects have failed or changed direction due to these difficulties. An example is Argo AI, which went bankrupt despite a billion-dollar investment,” he added.
According to a study published by Artificio last December, the perceptual failure of autonomous driving occurs due to two main factors.
First, a lack of perceptual understanding of out-of-distribution stimuli. And, second, a lack of high-quality benchmarks for such out-of-distribution scenarios.
“In many cases, autonomous vehicles are trained with real or synthetic data in almost perfect conditions of sun or with few people, with perfectly functioning traffic lights, well-paved roads, law-abiding drivers, and attentive pedestrians,” according to the study. “However, when things go wrong and the cars must navigate through fog, a crowd in a parade, dogs running (or playing) in the street without a leash, or checking for scooters in the side lanes, perceptual inference in many of these systems begins to falter.”
Artificio’s Proposal: Testing autonomous driving in extremely adverse conditions
In his mind, Deza saw his own backyard as an ideal testing ground.
Artificio’s founder suggests the AV industry first collect more data from developing countries, where driving patterns are different and more challenging, such as in Lima. This would help to better train computer vision systems used by autonomous cars.
“If an autonomous vehicle can easily navigate and identify every object and obstacle from video footage of routes in Lima, Hanoi, or Mumbai, then making perceptual inferences in cities like San Francisco, London, and Beijing should be trivial,” Deza said.
Additionally, Artificio proposes that the training of autonomous vehicle computer vision systems be conducted outside the city where they will be deployed. In machine learning, this is called Adversarial Training. This method trains systems with adverse and uncommon data, increasing the likelihood that neural networks will perform well in expected and unexpected situations.
Data collection on Peruvian streets
If you’ve visited Lima, you may know that the metropolitan area of 11 million inhabitants can be chaotic to get around in. According to a recent TomTom traffic study, Lima leads the ranking of cities in Latin America with the worst traffic, surpassing other large metropolises like Bogotá and Mexico City. Additionally, it ranks fifth in the index of cities with the worst traffic congestion in the world, only edged out by Milan, Toronto, Dublin, and London.
Due to this reality, between December 2023 and March 2024, Artificio collected data in various Peruvian cities, including Lima, Cusco, and Cajamarca, using dash cams and GoPro cameras mounted on vehicles.
“Data collection is simple, but analyzing and labeling it is a more complex and laborious process,” explained Deza. “Our goal is to benchmark with annotated data to evaluate model accuracy.”
The Artificio team has faced unique challenges, such as identifying uncommon vehicles in Peru, including mototaxis and ice cream vendor carts, which are not included in current model categories.
“This study is teaching us that we need new categories of objects for different geographical contexts, as the situations in India, for example, will be different from those in Peru,” added Deza.
The goal, he said, is to find fundamental principles that can be generalized to all cities. “If we achieve that, other companies will benefit as well.”
An innovative approach with Neural AI for object classification
Unlike current artificial intelligence models, which rely on large amounts of data and computational resources (like GPUs and terabytes of data) to achieve object classification, the Neural AI technology developed by Artificio aims to achieve the opposite.
Deza explained, “Our technology seeks to use the least amount of data and resources possible to create a system equal to or better than existing models, like Detectron2, YOLO, or RetinaNet. This is crucial for several reasons, including cost-benefit and ethical data handling.”
The idea behind the company’s proprietary NeuroAI foundational models, according to the CEO, is that autonomous vehicles can query any object they see on the road and find the most similar, already classified object to resolve the conflict in real-time.
Artificio’s long-term vision
Of course, there’s a long way to go for Artificio to compete with — and possibly help out — the largest AV companies. Its next step, Deza said, is to commercialize its platform, allowing any student or engineer to upload and evaluate their models through a web application.
“We are building a platform where people can upload and evaluate their models, even without being part of a company, which democratizes access to these advanced technologies,” he said.
Additionally, Artificio plans to incorporate light detection and ranging (LIDAR) sensors into vehicles in Peru, further enriching its data platform and demonstrating these systems’ capabilities in extreme conditions.
“If a self-driving car can handle Peru, it can handle anywhere,” said Deza.
Despite its setbacks, he envisions a promising future for autonomous driving, although he acknowledges that many challenges still must be overcome. “I believe that by 2030, autonomous cars should be operational globally, but the challenge is for companies to survive until then due to high costs and slow adoption.”