Gary Marcus (2024-11-09). CONFIRMED: LLMs Have Indeed Reached a Point of Diminishing Returns.
Yesterday I shared a link to a post suggesting that the next version of OpenAI’s frontier model (codenamed ‘Orion’) may not be much smarter than GPT-4.
Today’s link is from Gary Marcus, talking about the same news but from a different perspective. Marcus has been talking about the probability that LLMs aren’t the AI architecture that will get us to AGI, and in this he is probably right (not that I’m qualified to weigh in, but he is).
Sky high valuation of companies like OpenAI and Microsoft are largely based on the notion that LLMs will, with continued scaling, become artificial general intelligence. As I have always warned, that’s just a fantasy. There is no principled solution to hallucinations in systems that traffic only in the statistics of language without explicit representation of facts and explicit tools to reason over those facts.
In case it’s not clear, there are different perspectives on LLMs and why they matter. There are climate change concerns, ethics concerns, technology concerns, etc. Which is why the following statements can all be true at the same time:
- LLMs aren’t the right architecture to get us to AGI
- LLMs aren’t going to get much smarter
- LLMs are already incredibly smart
- LLMs are bad for the environment
- LLMs still have enormous potential for growth
And when OpenAI suggests that their next models probably won’t represent as much of a leap in intelligence as we’ve come to expect, this says nothing about the mundane utility of today’s models.
From my perspective, you can stop development of frontier language models tomorrow, and we’ll still have 10 years of work just to change our systems and processes to take them into account. And there are plenty of options for companies wanting to enhance model features, without concomitant increases in intelligence. For example:
- Make them cheaper
- Make them faster
- Make them smaller
- Give them more memory
- Make them better at the things they’re not currently very good at (e.g. planning, reasoning, etc.)
- Make them work together (i.e. as agent swarms)
As you can see, there’s still enormous potential for development and improvement, without increasing intelligence. If the next version of OpenAI’s frontier model doesn’t demonstrate as much of a leap as the move from GPT-3 to GPT-4, I’m not concerned.
I still haven’t reached the limit of what I can do with today’s models.