Bindu Reddy@bindureddy
The Emergent Abilities of LLMs Could Be A Mirage!
The best paper award in NeurIPs 2023 went to a paper claiming that the emergent abilities of LLMs could be a mirage!
The paper (link in alt) asserts that emergent abilities appear due to the researcher’s choice of metric rather than fundamental changes in model behavior with scale.
Let's understand some terms before getting into the details.
Emergence is a phenomenon whereby new properties may materialize in systems as their complexity increases. These properties can't be predicted from a precise quantitative understanding of the system’s microscopic details.
Emergent properties of LLMs are abilities that are not present in small models that manifest themselves in larger models (Sharpness), and their performance on specific tasks can emerge quite unpredictably and abruptly at scale (Unpredictability)
A lot of drama around LLMs taking over the planet involves emergence. Researchers argue that some scary emergent properties like free will and consciousness can magically manifest themselves in LLMS, and therefore, we have to pause, ban, and regulate AI research.
The paper excellently and credibly argues that the LLMs DO NOT possess emergent abilities - by this, they mean that there isn't anything sharp or unpredictable about them.
They show that smooth, continuous, predictable changes in model family performance appear sharp and unpredictable based on the choice of metric. So bigger models naturally and smoothly are more performant; there isn't some sharp jump in performance.
They also find that for non-linear metrics, smaller models are more performant than previously reported when they add additional test points to increase the resolution of the benchmark.
Overall, the point of the paper is that LLM behavior is NOT unpredictable, and in fact, larger LLMS are predictably more performant than larger ones.
In other words, there is no scientific reason to believe that LLMs can magically become supervillains one day.
If there is one paper you should read about LLMs, I recommend this one!
TLDR: There is no magic voodoo happening with LLMs; it's all math and statistics... as all deep learning is
