Joe Cole - e/acc

This is what I want to say: Been playing w GLM 5.2 for the past week and it’s legit. If I can get this in a box sitting on my desk, I’m not sure I need much else But this is probably what’s actually true: Can imagine a guy running on a Pentium PC for the first time and being like, wow, I don’t need anything else. But actually we invent all sorts of cool new things we can use compute. So each frontier/adjacent LLM is cool but we’ll always want to use the next, better one

Its been no secret that benchmarking has long been outpaced vastly recently by model improvements, but the infrastructure around it breaking means raw performance becomes more unwieldy and expensive to measure. The slowdown in infrastructure to benchmark/eval effectively is the hinderance to most enterprise AI adoption. Enterprise AI adoption strays away from much model post-training efforts not only because of perceived high cost/know-how constraints, but because the act of post-training itself is highly subjective in how it translates to business KPIs in lack of custom evals. That much of data markets remains a game of telephone in translating task realism from contrived data producer —> post-training regime —> unwieldy benchmark —> real world application means that post-training in today’s regimes with today’s benchmarks scarcely adapts one’s data to actually relevant processes. In a period where benchmarks break constantly, it is useful to explore certain approaches of certain benchmarks whose construction behavior we should encourage. To name a few: @cognition FrontierCode's FP/FN analysis @harvey LegalBench's model kickoff prompts that avoid tasks sounding like instructing someone with amensia @OpenAI Healthbench's "consensus" mechanisms where extreme rigor is placed on aligning LLM as a judge with real world expert’s opinions (+ -10/+10 reward rubric grading) @AnthropicAI BioMysteryBench's superhuman quesiton generation via controllable properties of data And all of the benchmarks who've started listing infrastructure specs, as infrastructure specs become larger determinants of model performance at long horizons. Altogether, multidimensionality of unverifiable verification approaches, as well as overtures from the cost-latency side of the Pareto curve threaten the validity of most benchmarks today. Just as AI engineering become an overnight skill in 2023, eval creation shall become one in the latter half of this year as a subset of that.

When tokens go down 90% by the end of the year and then another 90% next year, everyone's opinions on artificial general intelligence and superintelligence are going to change radically I'm currently on an unlimited GLM 5.2 bittensor subnet and I can tell you your behavior changes radically when token prices plummet

really interesting research direction imo, claude and gpt have very similar benchmark scores on pretty much everything, but yet are so different in the way they interact with humans

No idea what Thinking Machines is working on, but this line goes hard.

What do we do at @thinkymachines: Personalization/sovereignty, Human Participation, Decentralization. Democratize AI and make it useful for people. All three of them reduce society's dependence on centralized AGI companies (including ours when we get important), and that is a future worth aiming for. You've seen a preview of this with Tinker, Interaction models and our research openly published on Connectionism. A **lot** more to come very very soon...

no wonder Thinking Machines has the most immaculate vibes of any frontier lab/neolab Hayekian AI alignment: local knowledge, decentralized power, pluralism, and humans shaping the machines -- not the other way around

Today we share the worldview behind our mission. Human values don't average out. Local knowledge can't be centralized. The good future has many AIs, raised in different places, shaped by the people they serve, disagreeing with each other the way we do. thinkingmachines.ai/blog/the-futur…

One DGX Spark. Qwen3.6-35B 64 users. 700+ tok/s. 32,768 tokens in 54 seconds. 38W Each user has their own prompt and their own KV cache, and vLLM batches every active stream through the GPU each step. Recipe → github.com/Weschera/spark… @NVIDIAAI

RL can't teach an LLM to solve problems it never solves. Robot policies suffer from exactly the same limitation. The fix turned out to be almost embarrassingly simple: train the policy with diffusion noise during pre-training. That's it. The policy covers a much wider action distribution, RL finally has somewhere to search, and we fine-tune VLAs on real robots in under an hour. Introducing TMRL. 🧵(1/9)

We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability. We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as a leading coding eval. openai.com/index/separati…

We used ultrasound to let you talk without making any sound. After just a month of collecting data, our model is already approaching existing silent speech modalities. We were surprised to find that it generalizes to unseen participants as well! (1/n) x.com/vadi_ms/status…

It's time to get out of stealth 👋 Today, we are launching @monogram_ai and announcing our $40m seed round led by DST and Lux Capital. Monogram is the first AI app that was built around a visual interface, from the ground up. We created a technology that generates an entire user interface on the fly, in just a few seconds. Ask anything, and instead of staring at a wall of text, you get an interactive visual response.

I'm excited to announce the most ambitious recreation of Bloom's 2Sigma study of the last 40 years. It's funded by @reedhastings and staffed by a team of 20 of the best educators in this country. Our education team's goal is to show the largest academic gains in one year ever recorded, and we'll publish our results even if we fail. Recently, @jwdanner introduced me to @reedhastings. Most people know Reed co-founded Netflix. Fewer know he has been one of the driving forces behind improving education for the last twenty years. When Reed pitched me on recreating Bloom's famous 2-sigma problem, I felt an overwhelming sense of hope for education. Over the last few months we have moved at breakneck speed to assemble an exceptional team. Someone recently described it to me as the "Avengers of education." We are testing one question with the rigor it deserves: can elite one-on-one tutoring reproduce the largest learning gains ever measured in a controlled study? We will work with researchers from Stanford, Brown, Cornell, and other leading institutions, and we intend to be the most transparent research group in the field. That means publishing our methods, our benchmarks, and our results, whatever they show. We will invest up to $100,000 per year per student to give them the best education on the planet. If it works, the data and methods can help educators and technologists recreate these outcomes for every child. We are actively hiring tutors, engineers, and operations people to help us climb this mountain.

“The dirty secret in AI is that everything is a data and an eval problem. The best models have the best data and best internal benchmarks. The mid ones buy a lot of data, not the best, and hillclimb public benchmarks. (you need a lot of compute too)” – Stanford CS Professor

David Friedberg says Anthropic asked big pharma for their data and nearly everyone said no "There's been an effort by Anthropic to sign up life sciences companies to contribute to a new life sciences focused model. They're approaching these large companies with large proprietary data sets and saying, if you share your data, we will give you early access, some sort of proprietary value. Sign this NDA and you can participate with us." "I think nearly everyone I've spoken with has woken up to the fact that they are trying to commoditize everyone's business. If all of the tens of billions of dollars you have invested in experiments and product development, and you've generated all of this proprietary data along the way, that data is a true asset of your organization. It's an asset that you've spent billions of dollars developing." "And by handing it over to a model company to then combine with other people's data, you are commoditizing the one core differentiation that you have. And so everyone is largely saying no." "I think what everyone's realizing is they're better off developing their own weights and their own models using either an open source basis or there might be some intermediary business model that evolves."

Even before the agentic revolution, prompting tricks stopped being very valuable, as our research has shown. The best approach to AI right now is to clearly specify your goals, your output, what "good" & bad look like, how to test the results... (yes, this is just management)