Celeste HC

seems much more lightweight than I expected... capability enhancement likely reflects a dual contribution from both harness engineering and scaling laws 🤔

this relationship between emotion behaviours and domain-specific performance of LLMs might be an inspiration for our later attempts🤩

our recent attempts on agentic frameworks reminded me that perhaps we should return agent-core LLMs and memory systems to a more naive stage. namely, less knowledge SFT, more structured memory format, and overwriting inherent memory instead of expanding it.

I'm addicted to emphasizing the importance of something by laying some groundwork. it might becomes a bad habit because recently people seem to have lost the patience and interest to complete this process of divergence and convergence 😢

omg glad to find we were noticed by domain-specific practitioners 🤣🙏

I was struggling choosing the most proper ending for this work as well. forks on my way grew in front of me like a huge decision tree. I gradually felt crucial that I cannot assume the result first and then constantly align my prejudices with it. whenever 😌

finally completed all experiments this round. involved phases were completely reconstructed--some became more logical, some were aborted. better than expected but partially deviated from the initial intention. I can't tell it is a relief or relinquishment. make it exist first.

LLM-as-a-Fuzzy-Judge: Fine-Tuning Large Language Models as a Clinical Evaluation Judge with Fuzzy Logic. arxiv.org/abs/2506.11221

another incredibly underrated paper: Thinking Like Transformers (Weiss et al, 2021) presents RASP: a programming language that compiles to transformer *weights*. can implement sort(), bincount(), etc. seems important. why don't interpretability people care about this?

@sthuyan not just about moving on. this is more about how to become a complete individual 🫡

@CeliaShu1024 Move on

@sthuyan 👌

@CeliaShu1024 Pokemon go!

Beyond the perhaps superficial semantic distinction between "reasoning" and "pattern matching", there is a fundamental gap in the practical capabilities and behavior of these systems. You don't create an invention machine by iterating on an automation machine.

How much do language models memorize? "We formally separate memorization into two components: unintended memorization, the information a model contains about a specific dataset, and generalization, the information a model contains about the true data-generation process. When we completely eliminate generalization, we can compute the total memorization, which provides an estimate of model capacity: our measurements estimate that GPT-style models have a capacity of approximately 3.6 bits per parameter. We train language models on datasets of increasing size and observe that models memorize until their capacity fills, at which point “grokking” begins, and unintended memorization decreases as models begin to generalize."

We discovered faster way to compute product of matrix by its transpose! This has profound implications for data analysis, chip design, wireless communication, and LLM training! paper: arxiv.org/abs/2505.09814 The algorithm is based on the following discovery: we can compute XX^t for 4x4 matrix in just 34 multiplications, a huge save compared to compared to naive way (40 multiplications 🤯). We can apply this algorithm to any m x n matrix X (with n, m >= 4) by dividing it into 16 blocks X_1, ..., X_16. - Estimated energy save: 5-10% ✅ - Estimated time save: 5% ✅ The discovery was made by combining Machine Learning-based Search and Combinatorial Optimization. We used RL to sample bilinear expressions. We then used combinatorial solvers (Gurobi) to enumerate relations between these expressions and combine these expressions together into one algorithm for XX^t. One way think of it is modification of AlphaTensor approach - We reduced the action space by a factor of a million (x1000000) at the expense of relying on combinatorial solvers. The matrix XX^t is used everywhere: - Data Analysis: linear regression - Finance: covariance matrix for asset returns - LLM training: Muon, SOAP, Shampoo - Wireless Communication: 5G, MIMO channel capacity This operation is performed trillions of times every minute globally. Imagine if we can save 5% of energy used for these computations! Coauthors: Yushun Zhang @ericzhang0410, Zhi-Quan Luo.

Try this prompt instead, works like magic 🪄 "Reflect on 5-7 different possible sources of the problem, distill those down to 1-2 most likely sources, and then add logs to validate your assumptions before we move onto implementing the actual code fix"

@sthuyan 🤣🤣🤣

@CeliaShu1024 Learn uncertain quantification and you will not fear of uncertainty any more.

Excited to share a big update on AMIE, our research AI doctor from @GoogleDeepMind and @GoogleAI Now, AMIE can “see” and interpret visual medical data within a diagnostic conversation. Yes, AMIE exceeded human doctors (PCPs) in many key metrics like diagnostic accuracy and multimodal reasoning in a simulated clinical exam (OSCE) study. But crucially, how we realised this upgrade was different from our previous works, which relied very much on domain-specific (pre- or post-) training. We demonstrate that, with no finetuning, the combination of (1) natively multimodal Gemini 2.0 Flash (2) domain-specific inference-time algorithm can result in a capable conversational diagnostic AI. Through this year long project, we felt the power of the evolving frontier foundation models in this important domain while lots of work still remains to be done. See the thread from @KhaledSaab11 to learn more: x.com/KhaledSaab11/s… Adding more details and pointers to deep dives from my colleagues below!

March 26, 2027. The Legend of Zelda live-action movie has a release date. It is being released by Nintendo in collaboration with Sony.

hoping that our efforts on evaluation system can really help medical LLMs make real improvement for the world. but right now, I am not sure if my dreams are just a bit too socialist and romantic 😌

Disagree; we're working on near-term AGI and we aren't primarily using LLMs