Romain Vernoux retweetledi

While intelligence *leverages* compression in important ways in representation learning, intelligence and compression are by nature opposite in key aspects.
Because intelligence is all about *generalization to future data (out of distribution)* while compression is all about *efficiently fitting the distribution of past data*. If you're optimal at the latter, you're terrible at the former.
If you were an optimal compression algorithm, the behavior policy you would develop during the first 10 years of your life (maximizing your extrinsic rewards such as candy intake, while forgetting all information that appears useless as per past rewards) would be entirely inadequate to handle the next 10.
Yi Ma@YiMaTweets
Based on our latest study: arxiv.org/abs/2311.13110, compression seems to be all there is in current AI systems, including GPT-4. The remaining question is: Can compression alone lead to general intelligence or even consciousness? My bet is a clear NO.
English





