Gating

@burkov we think we at Gating have a good formula for improvement.

People waiting for better coding models don't realize that the quadratic time and space complexity of self-attention hasn't gone anywhere. If you want an effective 1M token context, you need 1,000,000,000,000 dot products to be computed for you for each of your requests for new code. Right now, you get this unprecedented display of generosity because some have billions to kill Google while Google spends billions not to be killed. Once the dust settles down, you will start receiving a bill for each of those 1,000,000,000,000 dot products. And you will not like it.

Where are the household robots ... why don´t we have them yet? This is Danko from gating.ai about the future of deep learning

It seems like deep learning is hitting a wall. A paper in 2017 calculated why this will happen . The problem is with the variety that deep learning cannot support. The paper: link.springer.com/article/10.100…

@GaryMarcus I think, after this did not work, we should now start focusing on Gating: gating.ai

𝗘𝗳𝗳𝗼𝗿𝘁𝘀 𝗮𝘁 𝗺𝗮𝘀𝘀𝗶𝘃𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝘁𝗵𝗮𝘁 𝗳𝗮𝗶𝗹𝗲𝗱 𝘁𝗼 𝗽𝗿𝗼𝗱𝘂𝗰𝗲 𝗔𝗚𝗜 (partial list) GPT-Turbo GPT-4o o3 mini-high GPT 4.5 Grok 3 Claude 3.7 Sonnet Gemini 2.0 Flash Meta Llama 3.1 Sora …. 𝗘𝗳𝗳𝗼𝗿𝘁𝘀 𝗮𝘁 𝗺𝗮𝘀𝘀𝗶𝘃𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝘁𝗵𝗮𝘁 𝘀𝘂𝗰𝗰𝗲𝘀𝘀𝗳𝘂𝗹𝗹𝘆 𝘆𝗶𝗲𝗹𝗱𝗲𝗱 𝗔𝗚𝗜 [𝘯𝘰𝘯𝘦]

@ylecun @DrJimFan @RichardSSutton I don't think it is architectures that makes us primarily smart. It is something else related to inductive biases and levels of adaption. These principles we embodied in "Gating" (gating.ai).

Animals and humans get very smart very quickly with vastly smaller amounts of training data. My money is on new architectures that would learn as efficiently as animals and humans. Using more data (synthetic or not) is a temporary stopgap made necessary by the limitations of our current approaches.

It’s pretty obvious that synthetic data will provide the next trillion high-quality training tokens. I bet most serious LLM groups know this. The key question is how to SUSTAIN the quality and avoid plateauing too soon. The Bitter Lesson by @RichardSSutton continues to guide AI development: there’re only 2 paradigms that scale indefinitely with compute: Learning & Search. It’s true in 2019 at the time of writing, true today, and I bet will hold true till the day we solve AGI. incompleteideas.net/IncIdeas/Bitte…

@MoonlitMonkey69 @Algon_33 @ylecun Correct. In ML language, genes carry inductive biases for learning. Gating (gating.ai) has been designed with this principle in mind.

@Algon_33 @ylecun Humans are biologically heuristic - the structure of the brain from our genetic code that comes from many thousands of years of evolution. Our individual lives are more like fine-tuning, or RHLF. Our 'base model' is our DNA. In reality we are training on VASTLY more data.

As long as AI systems are trained to reproduce human-generated data (e.g. text) and have no search/planning/reasoning capability, performance will saturate below or around human level. Furthermore, the amount of trials needed to reach that level will be far larger than the amount of trials needed to train humans. LLMs are trained with 200,000 years worth of reading material and are still pretty dumb. Their usefulness resides in their vast accumulated knowledge and language fluency. But they are still pretty dumb.

@ylecun If you put a hand-held calculator and related technology to this graph, you will see very quickly super human capability in multiplying and dividing numbers. There is something in symbolic thinking, something very poweful.

@ylecun @Algon_33 In my opinion, the best way to go is described here: gating.ai

@Algon_33 That's why we need AI systems that learn from high-bandwidth sensory inputs (like vision) not just text. drive.google.com/file/d/1pLci-z…

@ylecun @Algon_33 I think this is wrong logic. These data are mostly noise, are repetitive and are to a high degree even garbage. You know how we need "clean data". Well, these data are the opposite of clean. You cannot do much with these data. Moreover, how about blind people?

How to find information about the gating technology: gating.ai

Here is one example why LLMs do not really reason but rather perform pattern recognition. I asked ChatGPT 4o to perform a parity function on the binary string below. The LLM miscounted the 1s in the string. It is off by one, which is enough to return an incorrect answer.