The self-driving car industry is facing public scrutiny. Despite setbacks, Ghost Autonomy partnered with OpenAI to make a bold proposition: integrating multimodal large language models (LLMs) into autonomous driving.
Ghost secured a $5 million investment from OpenAI’s Startup Fund, along with access to advanced artificial intelligence (AI) systems and resources. The company believes its platform allows for the combination of AI and advanced autonomous driving software into the next generation of vehicles.
Ghost Auto’s bold proposition
Ghost Auto says OpenAI’s investment will “accelerate ongoing research and development of LLM-based complex scene understanding required for urban autonomy.”
These models could be capable of handling complex driving scenarios. Ghost says it is actively testing these capabilities in its fleet. Moreover, it plans to bring other partners on board to integrate LLMs into the autonomy stack.
“Solving complex urban driving scenarios in a scalable way has long been the holy grail for this industry. LLMs provide a breakthrough that will finally enable everyday consumer vehicles to reason about and navigate through the toughest scenarios,” says Ghost founder and CEO, John Hayes.
Skepticism from experts
Despite Ghost’s bold plan, experts are raising concerns, including PhD candidate on law and data ethics at the University of Washington, Os Keyes. In conversation with TechCrunch, Keyes says Ghost is using ‘LLM’ as a “marketing buzzword.”
“Basically, if you take this pitch and replace LLM with ‘blockchain’ and sent it back to 2016, it would be just as plausible — and just as obviously a boondoggle [unnecessary].”
Meanwhile, Mike Cook, a senior lecturer at King’s College London whose research focuses on computational creativity states that LLMs aren’t a “solved science”. He says in computer science, there’s no such thing as a silver bullet and it would be irresponsible to use it as such.
“There’s simply no reason to put LLMs at the center of something as dangerous and complex as driving a car. Researchers around the world are already struggling to find ways to validate and prove the safety of LLMs for fairly ordinary tasks like answering essay questions.
“The idea that we should be applying this often unpredictable and unstable technology to autonomous driving is premature at best — and misguided at worst,” Cook says.
Current state of self-driving industry
While Ghost’s ambitious plan sounds good on paper, public skepticism around self-driving cars is at an all-time high.
General Motors’ autonomous vehicle subsidiary, Cruise, recently initiated layoffs. This strategic move likely links to a hit-and-run accident involving a pedestrian, a human-driven vehicle, and a self-driving car named Panini.
ALSO READ: Cruise adjusts course amid layoffs and operational pauses
Other cons of self-driving cars often cited include the lack of human interaction, regulatory changes, widespread job losses, as well as the vehicles’ susceptibility to hacking and cyber threats.
On the flip side, a recent study shows how self-driving tech could boost the UK’s autonomous industry by £66 billion (US$80.8 billion) within the next two decades.
The Society of Motor Manufacturers and Traders (SMMT) says this technology could save around 4,000 lives by preventing 60,000 serious accidents. It could also create around 340,000 jobs, and lead to lower insurance premiums.
The bigger picture
Ghost is going beyond just integrating AI into self-driving vehicles; it aims to reinvent how self-driving cars perceive and interact with the environment – a move set to revolutionize the industry.
There will be challenges. Language models still make factual errors, and the models needed for self-driving cars are far from commercial readiness or deployment.
ALSO READ: New US Executive Order on the safe use of AI labeled terrible and risky
Pros and cons of LLMs aside, the future is autonomous and the science behind training self-driving cars is ever-evolving. A recent example is Waymo’s Waymax, a simulator designed for this purpose.
Unlike traditional simulators that rely on predefined agents scripted to behave in specific ways, Waymax analyzes observed behaviors using AI. It then develops robust, scalable AV systems which enables researchers to train models more realistically.
Despite these hurdles, Ghost’s vision points towards a transformative future in autonomous driving.
About the author
Cheryl has contributed to various international publications, with a fervor for data and technology. She explores the intersection of emerging tech trends with logistics, focusing on how digital innovations are reshaping industries on a global scale. When she's not dissecting the latest developments in AI-driven innovation and digital solutions, Cheryl can be found gaming, kickboxing, or navigating the novel niches of consumer gadgetry.
