Avijit Ghosh

Today, @evaluatingevals is introducing Every Eval Ever, a unified, open data format and public dataset for AI evaluation results.

What????? 🤦‍♂️

So happy to see Every Eval Ever (@evaluatingevals) take off! This is a big vote of confidence, and we really hope that we, as a community of eval practitioners, can move towards open standards that unlock scientific rigor and reproducibility. Thanks @mercor_ai !

@pfau @maxvonhippel We don’t know what we don’t know and we never will if we force people to use LLMs with limited conceptual solution spaces!

The degree to which AI research at the big labs has almost entirely been reduced to hill climbing is actually an aberration and not reflective of the rest of science at all. Ironically this means AI research is probably the easiest branch of research to automate.

Evaluation research? There's no place like evalEval

This is a good time to mention that the latest versions of both Claude and ChatGPT detect the hidden phrases and warn you of prompt injection, so I’m curious as to how this happened anyway/which LLMs were still susceptible

@jackclarkSF @AnthropicAI I have a maybe naive question. It seems like the entire Anthropic economic index stack is built on top of Clio data, which, albeit cool, is a small subset of AI usage data writ large. How’s the team generalizing the findings to the entire economy by only looking at Claude data?

I'm scaling the economic research function here @AnthropicAI to meet the challenge of powerful AI. This team today produces the best data in the industry via the Anthropic Economic Index + recent work on job exposure to AI. We have many very ambitious plans in the works. Join!

We at @huggingface are fortunate to have a unique vantage point on the state of open source AI development. We finally wrote down our observations, from both our own research and that of our peers who have done excellent work investigating the open ecosystem with Hugging Face hub data and beyond. Presenting the State of Open Source on HF: Spring 2026 edition! Extremely juicy blog post with lots of insights to continue following this year 🤗 Work with @frimelle, @YJernite and @IreneSolaiman!

Always a hoot reading Georgia’s takes! Case in point: While Rosie the dog’s cancer treating MRNA vaccine made with LLMs+Alphafold went viral, several domain scientists on here have pointed out both novelty issues and the structural problems with generalizing this to large scale human trials. Very glad our paper is now peer reviewed! Please give it a read.

@crystalsssup Hopefully Nano banana improves enough to make this possible! It still struggles with spatial alignment within images (“move the creat halfway to the inner surface of the envelope liner so that it is half hidden when the envelope is assembled” fails spectacularly)

This feature is going to replace TED-Ed sooner or later. Education in this era be like: if you're curious, you can learn anything from the best teacher—AI. It's also a very smart move by the Google team: Nano Banana generates the infographic, Veo 3 makes it move, TTS adds voiceover. Very strong combination.

@paulg Oooo L.A. Burdick mentioned 🥳

Brands I love: Lego, Leuchtturm, Oxford University Press, Pentel, Schöffel, Aqualung, Paradores, Staedtler, Birkenstock, Braun, Knoll, Patagonia, Herman Miller, Iittala, L.A. Burdick, Artemide, Aman, Thames & Hudson, Yeti, Rimowa, L.L.Bean, Timbuk2, Eschenbach, Ridge, Maui Jim.

Imagine if my little AGI robot knew how to put my “I have worn it once but it’s not yet dirty enough to launder” clothes on this purgatory chair 😍

Long post, because apparently many neither understand nor appreciate the intricacies of cancer research and think that pharmaceutical companies and regulators are holding back cures. Your immune system is constantly surveilling your body for both self and non-self recognition. It does this by checking the proteins expressed. If it finds something it doesn't recognize, it ramps up the inflammatory response and attacks. If it is actually non-self, great. If it is actual-self, that is autoimmune disease. Cancer occurs when cells acquire mutations that both 1) alter cell division and 2) cloak those cells from recognition by the immune system. If the second part doesn't occur, then the immune system will recognize that something is wrong and kill the tumor when it's just a few cells. At a high level, many modern cancer therapies are about getting the immune system to recognize the tumor and do all the work on killing it. And yes, it's quite easy to do this. But this is where the "safe and effective" line comes in. If you're treatment just cranks up the immune response in general, you start killing the tumor AND other things. If you give me a decent CAR-T at work, I have tools to boost it in ways that'll eliminate any realistic sized mouse tumor in 24-48 hours. The problem is that it's just a general immune overdrive and the cells start attacking everything. Okay, let's not send the immune system into a frenzy and just use the CAR-T, which is a T cell that has been edited with a protein that we know binds to a protein that the cancer expresses. The hope is, if the cancer cells express X and we edit the T cells to explain anti-X, then they'll go and attack the tumor. But this is where specificity and selectively come in. Your body expresses thousands of different proteins. Sometimes they look very much like each other even if they are different. Say there is a protein called XX expressed in your heart, it shares 99.9% homology (likeness) with X. We inject X-CAR-T cells, they go and kill the tumor, everything looks great. But a few binded to XX in the heart and started inflamming the heart wall. Not good to the point of unacceptability. This step gets especially hard when working in animals because mice and dogs or whatever have different proteins than humans! When we inject human tumors and human CAR-T cells into mice, they are NOT encountering the same proteins (or cytokines, hormones, other immune cells, etc.) that they would in an actual human body. This is just a brief explanation or some of the considerations that go into oncology. Here's another: we monitor experimental mice for max a few months. Even non-experimental mice have a lifespan of ~1.5-2 years. Meanwhile, you want your parent/spouse/child to be in remission for 5, 10, 15+ years! In fact, one of the main outcomes for assessing human cancer treatments is the "5-year survival rate." Mice and elderly dogs don't live for five years!! So yes, scientists and pharmaceutical companies have tools to easily kill tumors. What is hard is developing therapeutics that are BOTH safe AND effective in actual humans **relative** to current standards of care (e.g. a cutting edge treatment isn't "better" if it has a strong response in the first year but a lower 5-year overall survival, etc.) And this is where regulators come in. I expressed multiple times in the comments that I, generally, wish they weren't as risk averse and that I support "Right To Try" laws. But you need to appreciate that regulators **are** in a DIFFICULT position. An analogy: It's proven that nuclear energy is by far the safest form of energy per KwH energy produced. Yet a few high profile accidents, a couple of which didn't even kill anyone, have poisoned a large segment of the population and several nations against nuclear energy. Even though it's safe and a reliable form of carbon-free electricity!!! Now think about that relative to cutting edge medicines.

Sorry to be the downer because this is an impressive story in some senses. But it is ~trivially easy to make a single mRNA vaccine. It's not hard. I cure mice of various cancers with various therapeutics all the time. I've made mice lose more weight in a month than tirzepatide does in a year. What is hard and expensive is proving its BOTH safe AND effective **in a randomized and controlled study in humans** while ALSO manufacturing it at clinical scale and grade. I am happy for this man and his dog. It is impressive. But y'all are overhyping it.

Unusual open data move by a major AI labs: StepFun releases the general SFT training set of Stepfun-Flash.

This is wild. theaustralian.com.au/business/techn…

Seeing the worldwide demand we are kicking off global applications for Hugging Face Builders! If you're passionate about open AI and love bringing people together, this is your invitation to lead ✉️ Learn more about the program and apply to become a Builder ➡️

👀

Next step: Open sourcing this UX stack 😈 who’s building a nice wrapper that does responsive UX where we can swap out the models in the back end?

A welcome development! (I’ve been on an anti chatbot rant lately)