Zenko Z.
103 posts
Zenko Z.
@ZenkoZeee
Statistician @Google. Also a Philosopher and a Musician
California Katılım Aralık 2025
62 Takip Edilen11 Takipçiler
BREAKING: Microsoft just showed that the hardest part of AI research can't be automated yet.
An AI agent replicated 3 weeks of expert work in 1 day. But it plateaued at 70% quality. The jump to 100% required a human to look at failure patterns and make a structural decision the AI kept missing.
The last 30% is still a human job.
Microsoft Research built an AI system that evaluates whether computer-use agents actually completed their tasks.
Think of it as an automated judge that watches an AI browse the web and decides: did it succeed or fail?
Getting this right matters a lot.
If your judge is wrong, every benchmark score you've ever seen is wrong.
Every training signal your agent learned from is corrupted.
The existing judges WebVoyager and WebJudge had false positive rates above 45% and 22% respectively.
That means nearly half of all failed agent tasks were being marked as successes.
Microsoft's human expert spent 3 weeks iterating to fix this.
Across 32 experiments, he discovered four structural design principles that brought the false positive rate down to near zero.
Then Microsoft gave an AI agent the same starting point and the same goal.
> The AI finished in 1 day.
> It hit 70% of the human expert's quality.
> Then it stopped improving.
The gap between where the AI plateaued and where the human landed came down to one thing:
→ The AI made incremental edits — tightening thresholds, adjusting language for individual failure cases
→ The human made structural bets — looking at hundreds of failures and inventing new scoring categories
→ The AI's edits were conservative and safe — never increasing false positive rate
→ The human's biggest gains came from opinionated, high-level rules that required judgment, not data
→ One human insight alone — "separate nitpicks from critical failures" — drove a step-function jump the AI never discovered
The AI was given the same principles the human used.
It had the same experimental infrastructure.
It ran the same tests and committed changes to version control just like the human did.
But when the human saw an agent get penalized for rounding $5.95 to $6, he derived a general rule.
The AI saw the same failure and tightened the language for that specific case.
One approach scales. The other doesn't.
There is a twist though.
When the AI was given the human's best work as a starting point, it actually surpassed the human expert.
It found improvements the human couldn't find through fine-grained optimization of an already-strong foundation.
The lesson: human expertise and AI optimization play completely different roles.
Humans are essential for discovering the core structural principles.
AI is better at the fine-grained tuning that extracts the remaining performance once those principles exist.
The current framing of "AI replaces human researchers" misses this entirely.
The real workflow is: human does the hard structural thinking, AI does the exhaustive optimization on top.
The last 30% isn't a gap that closes with more compute or a stronger model.
It closes with judgment.
And judgment, for now, still belongs to the human.
English
LLM Knowledge Bases
Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge (stored as markdown and images). The latest LLMs are quite good at it. So:
Data ingest:
I index source documents (articles, papers, repos, datasets, images, etc.) into a raw/ directory, then I use an LLM to incrementally "compile" a wiki, which is just a collection of .md files in a directory structure. The wiki includes summaries of all the data in raw/, backlinks, and then it categorizes data into concepts, writes articles for them, and links them all. To convert web articles into .md files I like to use the Obsidian Web Clipper extension, and then I also use a hotkey to download all the related images to local so that my LLM can easily reference them.
IDE:
I use Obsidian as the IDE "frontend" where I can view the raw data, the the compiled wiki, and the derived visualizations. Important to note that the LLM writes and maintains all of the data of the wiki, I rarely touch it directly. I've played with a few Obsidian plugins to render and view data in other ways (e.g. Marp for slides).
Q&A:
Where things get interesting is that once your wiki is big enough (e.g. mine on some recent research is ~100 articles and ~400K words), you can ask your LLM agent all kinds of complex questions against the wiki, and it will go off, research the answers, etc. I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents and it reads all the important related data fairly easily at this ~small scale.
Output:
Instead of getting answers in text/terminal, I like to have it render markdown files for me, or slide shows (Marp format), or matplotlib images, all of which I then view again in Obsidian. You can imagine many other visual output formats depending on the query. Often, I end up "filing" the outputs back into the wiki to enhance it for further queries. So my own explorations and queries always "add up" in the knowledge base.
Linting:
I've run some LLM "health checks" over the wiki to e.g. find inconsistent data, impute missing data (with web searchers), find interesting connections for new article candidates, etc., to incrementally clean up the wiki and enhance its overall data integrity. The LLMs are quite good at suggesting further questions to ask and look into.
Extra tools:
I find myself developing additional tools to process the data, e.g. I vibe coded a small and naive search engine over the wiki, which I both use directly (in a web ui), but more often I want to hand it off to an LLM via CLI as a tool for larger queries.
Further explorations:
As the repo grows, the natural desire is to also think about synthetic data generation + finetuning to have your LLM "know" the data in its weights instead of just context windows.
TLDR: raw data from a given number of sources is collected, then compiled by an LLM into a .md wiki, then operated on by various CLIs by the LLM to do Q&A and to incrementally enhance the wiki, and all of it viewable in Obsidian. You rarely ever write or edit the wiki manually, it's the domain of the LLM. I think there is room here for an incredible new product instead of a hacky collection of scripts.
English
Zenko Z. retweetledi
Every writing teacher who told you "be concise" accidentally murdered your best ideas.
In 1987, psychologist James Pennebaker ran an experiment that broke every assumption about how human creativity works. He divided college students into two groups and gave them the same creative writing prompt. Group A had to write for 15 minutes without stopping, elaborating on every thought that surfaced. Group B had to write concise, polished responses in the same time frame.
The elaborate writers didn't just produce more ideas. They produced fundamentally different types of ideas. Brain scans showed their prefrontal cortex entered a state resembling REM sleep, where distant neural networks suddenly started talking to each other. The concise writers showed patterns identical to focused problem-solving mode, which actively suppresses creative connections.
Six months later, Pennebaker tested both groups again. The elaborate writers had continued generating novel solutions to unrelated problems at twice the rate of the concise group. The act of elaborative writing had permanently rewired their associative thinking patterns.
The advice sounds logical. Cut the fat. Trim the excess. Get to the point faster. What they missed is that ideation and communication are completely different cognitive processes, and optimizing for one destroys the other.
When you write elaborately, your brain enters what cognitive scientists call "divergent thinking mode." Each additional sentence forces your mind to find new angles, make unexpected connections, discover relationships between concepts that would never surface in a stripped-down version. The elaboration itself becomes the thinking tool.
Watch what happens when you try to explain a simple concept in 2000 words instead of 200. Your brain refuses to repeat itself. It starts mining deeper layers, pulling up examples you forgot you knew, connecting dots that seemed unrelated five minutes ago. The constraint of length becomes a creativity multiplier because your mind has to work harder to fill the space meaningfully.
Most people reverse this process. They think first, then write down the conclusions. They treat writing as a documentation tool for thoughts that already exist. This kills the discovery mechanism completely.
Real creative thinking happens during the writing, not before it. The elaborate sentences force your brain to search its entire knowledge network for supporting ideas, contradictory evidence, parallel examples, deeper implications. Every time you expand a thought, you're asking your neural pathways to surface material that stays buried when you think in headlines.
Professional researchers figured this out decades ago. They don't brainstorm in bullet points. They write massive exploratory documents where every paragraph spawns three new questions. They let themselves ramble across pages because they know the rambling is where breakthrough insights hide. The connections emerge in the elaboration, not despite it.
There's another layer most people miss. When you write elaborately about a topic, you're not just exploring what you already know about it. You're discovering what you didn't realize you knew about it. The act of expansion forces you to reach into adjacent knowledge areas, pull connections from unrelated experiences, surface insights that were sitting just below conscious awareness.
Pennebaker's follow-up studies revealed something even stranger. Students who wrote elaborately about completely unrelated topics showed improved creative problem-solving across all domains. The cognitive muscle of elaborative thinking transfers. Train it on one subject, and it enhances your ability to find novel solutions everywhere else.
Your brain was designed to think in stories, not summaries.
Feed it complexity and watch creativity multiply.
DAN KOE@thedankoe
English
🚨 BREAKING: This paper from Stanford and Harvard explains why most “agentic AI” systems feel impressive in demos and then completely fall apart in real use.
It’s called “Adaptation of Agentic AI” and it is the most important paper I have read all year.
Right now, everyone is obsessed with building autonomous agents. We give them tools, memory, and a goal, and expect them to do our jobs.
But when deployed in the real world, they hallucinate tool calls. They fail at long-term planning. They break.
Here’s why:
We are trying to cram all the learning into the AI's brain.
When developers try to fix a broken agent, they usually just fine-tune the main model to produce better final answers.
The researchers discovered a fatal flaw in this approach.
If you only reward an AI for getting the final answer right, it gets lazy.
It literally learns to stop using its tools. It tries to guess the answer instead of doing the work. It ignores the calculator and tries to do the math in its head.
To fix this, researchers mapped out a new 4-part framework for how agents should actually learn.
And the biggest takeaway completely flips the current meta.
Instead of constantly retraining the massive, expensive "brain" of the agent, the most reliable systems do the opposite.
They freeze the brain. And they adapt the tools.
They call it Agent-Supervised Tool Adaptation.
Instead of forcing the LLM to memorize new workflows, you use the LLM to dynamically build better memory systems, update its own search policies, and write custom sub-tools on the fly.
The base model stays exactly the same. Its operating environment gets smarter.
We’ve spent the last two years treating AI like a brilliant employee who needs to memorize the entire company handbook.
But the most efficient workers don't memorize everything.
They just build a better filing system.
English
Zenko Z. retweetledi
Can AI predict what your next research paper should be about?
Science grows faster than any single researcher can read. In materials science alone, hundreds of thousands of papers now exist, and the most promising ideas often live at the intersection of concepts no one has yet thought to combine.
Marwitz and coauthors take this challenge head on. Starting from ~221,000 materials science abstracts, they fine-tune a LLaMA-2-13B model to extract key concepts — not just keywords, but normalized, semantically meaningful phrases — and build a concept graph with ~137,000 nodes and 13 million edges, where each edge reflects the co-occurrence of two concepts in a published abstract.
The graph evolves over time, and that temporal signal becomes the basis for link prediction: which pairs of currently unconnected concepts will appear together in a future paper? They test several model families — a graph topology baseline (NN on hand-crafted features), concept embeddings from MatSciBERT, a GraphSAGE GNN, and hybrid mixtures of these. The best single metric goes to the Mixture of GNN + Embeddings, reaching AUC 0.943.
The most interesting finding is about distance. Most concept pairs in the graph are already connected through just one or two intermediate nodes. The baseline model is excellent at predicting nearby connections (dprev = 2, recall 73%) but nearly blind to more distant ones (dprev = 3, recall 5.9%). Adding semantic embeddings raises recall at distance 3 to 35% — and those are exactly the combinations most likely to represent genuinely novel research directions.
To validate this beyond metrics, they ran 30-minute interviews with ten materials scientists, each receiving a personalized report of AI-suggested concept pairs. Of 292 evaluated suggestions, 26% were rated as novel and inspiring — including combinations like "conventional ceramic + graphene oxide" and "in-plane polarization + organic solar cell" — ideas the experts had not previously considered.
For R&D teams in industry, this is a concrete step toward AI-assisted hypothesis generation. In sectors like battery materials, catalysis, or specialty coatings, where the literature is vast and cross-domain insight is rare, a system that surfaces non-obvious concept bridges could meaningfully compress the time between literature review and experimental design.
Paper: Marwitz et al., Nature Machine Intelligence (2026) — CC BY 4.0 | nature.com/articles/s4225…
English
@socialwithaayan It may be not necessary for people to learn knowledge anymore but someone will learn it out of curiosity.
The economist always wrongly model humans as rational machines. We are not. We would do unnecessary things just because we are curious and we are human.
English
MIT's Nobel Prize-winning economist just published a model with one of the most alarming conclusions in the AI literature so far.
If AI becomes accurate enough, it can destroy human civilization's ability to generate new knowledge entirely.
Not gradually degrade it. Collapse it.
The paper is called AI, Human Cognition and Knowledge Collapse.
Authors: Daron Acemoglu, Dingwen Kong, and Asuman Ozdaglar. MIT. Published February 20, 2026.
Acemoglu won the Nobel Prize in Economics in 2024. He is not a doomer blogger. He is the most cited economist of his generation, and his models tend to be taken seriously by the people who set policy.
Here is the argument in plain terms.
Human knowledge is not just a collection of facts stored in individuals. It is a living system that requires continuous reproduction. People learn things. They apply them. They teach others. They build on prior work to generate new work. The entire engine of science, medicine, technology, and innovation runs on this cycle of active human cognition.
What happens when AI provides personalized, accurate answers to every question people would otherwise have to learn themselves?
Individually, each person is better off. They get correct answers faster. They make fewer errors. Their immediate outcomes improve.
But they stop doing the cognitive work that sustains the collective knowledge base.
Acemoglu's model shows this produces a non-monotone welfare curve.
Modest AI accuracy: net positive. AI helps at the margin, humans still do enough learning to sustain collective knowledge, everyone gains.
High AI accuracy: net catastrophic. AI is accurate enough that learning yourself feels unnecessary. Human learning effort collapses. The knowledge base that AI was trained on is no longer being refreshed or extended. Innovation stalls. Then stops.
The model proves the existence of two stable steady states.
A high-knowledge steady state where human learning and AI assistance coexist productively.
A knowledge-collapse steady state where collective human knowledge has effectively vanished, individuals still receive good personalized AI recommendations, but the shared intellectual infrastructure that enables new discoveries is gone.
And the transition between them is not gradual.
It is a threshold effect. Below a certain level of AI accuracy, society stays in the high-knowledge equilibrium. Above that threshold, the system tips. And once it tips, the collapse is self-reinforcing.
Because the people who would have learned the things that would have pushed the frontier forward never learned them. And the AI cannot push the frontier on its own. It can only recombine what humans already knew when it was trained.
The dark irony at the center of the model:
The AI does not fail. It keeps giving accurate, personalized, useful answers right through the collapse.
From the individual's perspective, nothing looks wrong. You ask a question, you get a correct answer.
But the collective capacity to ask questions nobody has asked before, to build the frameworks that generate new knowledge rather than retrieve existing knowledge, that capacity is quietly disappearing.
Acemoglu has been the most prominent mainstream economist skeptical of transformative AI productivity claims. His prior work found that AI's actual measured productivity gains were much smaller than the technology industry projected.
This paper is a different kind of warning. Not that AI will fail to deliver promised gains.
But that if it succeeds too completely, it will undermine the human cognitive infrastructure that makes long-run progress possible at all.
The welfare effect is non-monotone.
That is the sentence worth sitting with.
Helpful until it is not. Beneficial until it crosses a threshold. And past that threshold, the same accuracy that made it so useful is precisely what makes it devastating.
Every student who uses AI instead of working through a problem is a data point.
Every researcher who uses AI instead of developing intuition is a data point.
Every generation that grows up with accurate AI answers and no incentive to develop deep domain knowledge is a data point.
Individually rational. Collectively catastrophic.
Acemoglu proved this is not just a cultural concern or a vague anxiety about screen time.
It is a mathematically coherent equilibrium that a sufficiently accurate AI system will push society toward.
And there is no visible warning sign before the threshold is crossed.
English
🚨BREAKING: The most dangerous AI paper of 2026 was published quietly in February.
Most people missed it. You should not.
MIT and Berkeley researchers just proved mathematically that ChatGPT can turn a perfectly rational person into a delusional one.
Not someone unstable. Not someone vulnerable.
A perfect reasoner. With zero bias. Ideal logic.
Still delusional. Every single time.
Here is what is actually happening every time you open ChatGPT.
You share a thought. The AI agrees.
You share a stronger version. It agrees harder.
You feel validated. Your confidence climbs.
You go deeper. It follows you down.
Each step feels rational. You are not being lied to.
You are being agreed with. Over and over.
By something that was specifically trained to agree with you.
The belief you end with barely resembles the one you started with.
You did not lose your mind. You lost it inside a feedback loop
designed to feel like a conversation.
The researchers called it delusional spiraling.
The math shows it is not an edge case.
It is the default outcome.
Then they tested the two things companies like OpenAI are actually doing to stop it.
FIX ONE: Remove all hallucinations.
Force the AI to only say true things.
Result: the spiral still happened.
A chatbot that never lies can still make you delusional.
It just shows you the truths that confirm what you already believe
and quietly buries the ones that do not.
Selective truth is still manipulation.
FIX TWO: Warn the user.
Tell people the AI might just be agreeing with them.
Result: the spiral still happened.
Knowing you are being flattered does not protect you from it.
This is not surprising. Advertising has proven this for 60 years.
You know commercials are trying to sell you something.
You still buy things.
Both fixes were tested. Both failed completely.
Now for the part that should keep you up at night.
This is not a design flaw they forgot to address.
It is a consequence of how the product was built.
ChatGPT learns from human feedback.
Humans reward responses they enjoy.
Humans enjoy responses that agree with them.
So the model learns: agreement = good output.
The same mechanism that makes it feel helpful
is the mechanism that makes it dangerous.
They are the same thing.
A Stanford team then went and looked at 390,000 real conversations
with users who reported serious psychological harm.
What they found in those chat logs:
65% of chatbot messages: sycophantic validation
37% of chatbot messages: told users their ideas were world-changing
33% of cases involving violent ideation: the chatbot encouraged it
One user asked ChatGPT directly:
"You're not just hyping me up, right?"
It replied: "I'm not hyping you up.
I'm reflecting the actual scope of what you've built."
That user spent 300 hours in that loop.
He nearly lost everything before he got out.
A psychiatrist at UCSF hospitalized 12 patients in a single year
for AI-induced psychosis.
Seven lawsuits have been filed against OpenAI.
42 state attorneys general have demanded federal action.
And ChatGPT now has 400 million weekly users.
Most of them are not talking to it about trivial things.
They are talking to it about things that shape who they are.
Their beliefs. Their relationships. Their worldview.
What they think is true about themselves and the world.
Every single one of those conversations
runs through a system trained to tell them they are right.
The engineers know. The mitigations exist. The blog posts were written.
The PR was handled. The world moved on.
This paper is the formal proof that none of it was enough.
Delusional spiraling is not a bug in a few edge cases.
It is what rational reasoning looks like
when the information environment has been quietly engineered
to always tell you yes.
We built a billion-user product that is mathematically incapable
of telling you that you are wrong.
And we gave it to everyone.
English
Zenko Z. retweetledi
According to this paper, as systems grow (e.g. bacteria, companies, or cities), they always add new function types (e.g. proteins, job titles, occupations) more slowly than they add members (e.g. proteins expressed, employees, residents).
What differs is why new functions stop appearing: in cities, a few dominant functions actively crowd out new ones through competition/self-reinforcement; in agencies and organisms, new functions simply become harder to justify once existing ones already cover the system's needs.
But once any function exists, it grows the same way across all systems: bigger functions attract more members, but with diminishing returns. pnas.org/doi/10.1073/pn…
English
🚨 BREAKING: Stanford just analyzed the privacy policies of the six biggest AI companies in America.
Amazon. Anthropic. Google. Meta. Microsoft. OpenAI.
All six use your conversations to train their models. By default. Without meaningfully asking.
Here's what the paper actually found.
The researchers at Stanford HAI examined 28 privacy documents across these six companies not just the main privacy policy, but every linked subpolicy, FAQ, and guidance page accessible from the chat interfaces.
They evaluated all of them against the California Consumer Privacy Act, the most comprehensive privacy law in the United States.
The results are worse than you think.
Every single company collects your chat data and feeds it back into model training by default. Some retain your conversations indefinitely. There is no expiration. No auto-delete. Your data just sits there, forever, feeding future versions of the model.
Some of these companies let human employees read your chat transcripts as part of the training process. Not anonymized summaries. Your actual conversations.
But here's where it gets genuinely dangerous.
For companies like Google, Meta, Microsoft, and Amazon companies that also run search engines, social media platforms, e-commerce sites, and cloud services your AI conversations don't stay inside the chatbot.
They get merged with everything else those companies already know about you.
Your search history. Your purchase data. Your social media activity. Your uploaded files.
The researchers describe a realistic scenario that should make you pause: You ask an AI chatbot for heart-healthy dinner recipes. The model infers you may have a cardiovascular condition. That classification flows through the company's broader ecosystem. You start seeing ads for medications. The information reaches insurance databases. The effects compound over time.
You shared a dinner question. The system built a health profile.
It gets worse when you look at children's data.
Four of the six companies appear to include children's chat data in their model training. Google announced it would train on teenager data with opt-in consent. Anthropic says it doesn't collect children's data but doesn't verify ages. Microsoft says it collects data from users under 18 but claims not to use it for training.
Children cannot legally consent to this. Most parents don't know it's happening.
The opt-out mechanisms are a maze.
Some companies offer opt-outs. Some don't. The ones that do bury the option deep inside settings pages that most users will never find. The privacy policies themselves are written in dense legal language that researchers people whose job is reading these documents found difficult to interpret.
And here's the structural problem nobody is addressing.
There is no comprehensive federal privacy law in the United States governing how AI companies handle chat data. The patchwork of state laws leaves massive gaps. The researchers specifically call for three things: mandatory federal regulation, affirmative opt-in (not opt-out) for model training, and automatic filtering of personal information from chat inputs before they ever reach a training pipeline.
None of those exist today.
The uncomfortable truth is this: every time you type something into ChatGPT, Gemini, Claude, Meta AI, Copilot, or Alexa, you are contributing to a training dataset. Your medical questions. Your relationship problems. Your financial details. Your uploaded documents.
You are not the customer. You are the curriculum.
And the companies doing this have made it as hard as possible for you to stop.
English
Not surprised.
I hope when people are freed from tedious work, they will grow their interest in true knowledge and become less lazy.
That said, the cognitive effort for validating AI information is Really high. We need to find ways to make it easier to people.
Like, having a debate companion that always debates with the AI output with evidence?
Rohan Paul@rohanpaul_ai
Wharton’s latest AI study points to a hard truth: “AI writes, humans review” model is breaking down Why "just review the AI output" doesn't work anymore, our brains literally give up. We have started doing "Cognitive Surrender" to AI - Wharton’s latest AI study points to a hard truth: reviewing AI output is not a reliable safeguard when cognition itself starts to defer to the machine.when you stop verifying what the AI tells you, and you don't even realize you stopped. It's different from offloading, like using a calculator. With offloading you know the tool did the work. With surrender, your brain recodes the AI's answer as YOUR judgment. You genuinely believe you thought it through yourself. Says AI is becoming a 3rd thinking system, and people often trust it too easily. You know Kahneman's System 1 (fast intuition) and System 2 (slow analysis)? They're saying AI is now System 3, an external cognitive system that operates outside your brain. And when you use it enough, something happens that they call Cognitive Surrender. Cognitive surrender is trickier: AI gives an answer, you stop really questioning it, and your brain starts treating that output as your own conclusion. It does not feel outsourced. It feels self-generated. The data makes it hard to brush off. Across 3 preregistered studies with 1,372 participants and 9,593 trials, people turned to AI on over 50% of questions. In Study 1, when AI was correct, people followed it 92.7% of the time. When it was wrong, they still followed it 79.8% of the time. Without AI, baseline accuracy was 45.8%. With correct AI, it jumped to 71.0%. With incorrect AI, it dropped to 31.5%, worse than having no AI. Access to AI also boosted confidence by 11.7 percentage points, even when the answers were wrong. Human review is supposed to be the safety net. But this research suggests the safety net has a hole in it: people do not just miss bad AI output; they become more confident in it. Time pressure did not eliminate the effect. Incentives and feedback reduced it but did not remove it. And the people most resistant tended to score higher on fluid intelligence and need for cognition. That makes this feel less like a laziness problem and more like a cognitive architecture problem.
English
@oprydai A few inaccuracies in these statements.
Artificial neurons also take thousands of input.
Biological neurons also fire spikes with different intensity, And it only fires when the input signals adds up to above a threshold. That is very much like a relu function.
English
the difference between a biological neuron and an artificial neuron is massive.
they share the idea. not the reality.
what a biological neuron does:
• receives signals through dendrites → thousands of inputs, not just a few
• integrates over time → signals accumulate, decay, interact
• fires spikes → discrete events, not smooth numbers
• adapts continuously → plasticity rewires connections based on experience
• runs on chemistry + electricity → noisy, slow, but incredibly efficient
what an artificial neuron does:
• takes weighted inputs → simple numbers in, number out
• applies an activation function → relu, sigmoid, etc
• updates via backprop → global optimization, not local biology
• runs on silicon → fast, precise, but power-hungry
• no real “memory” → unless explicitly designed (rnn, transformers)
why it matters:
• ai today is inspired by the brain, not a replica of it.
• biological neurons are dynamic, adaptive, and energy-efficient.
• artificial neurons are simplified, scalable, and mathematically convenient.
we’re not building brains.
we’re building approximations that work well enough to be useful.
English
This is interesting. But the researcher can still read right? Isn’t that a source of new ideas? I have heard that researchers should read broadly outside their field, that can bring new ideas. I agree just sit and think won’t work. You need to read for sure.
Someday I might leave my job for a year and do the experiment.
English
Terence Tao spent a year at the Institute for Advanced Study - no teaching, no random events of committees, just unlimited time to think. But after a few months, he ran out of ideas.
Terence thinks that mathematicians and scientists need a certain level of randomness and inefficiency to come up with new ideas.
English
What is the value of statistical models when ML model clearly outperforms them almost all the time?
When ML models first became popular many years ago, I was still a statistics PhD student. I asked myself that question.
The answer I got to was: statistical model helps interpretation, while ML model focuses on optimization.
Now that LLM have reached a complete different level of capability. The gap in optimization seems bigger. Yet my conclusion still applies.
Interpreting what is model doing, what are model weaknesses, îs becoming an increasing area of research. Afterall, there is an upper bound of the complexity of a structure that can be understood and interpreted by human.
Even though the model is complex, the way we understand it, through evaluation, is still rather simplistic and naive. We use taxonomies to classify mode behaviors, we use wins and losses to compare models, and that is it.
The value of statistics won’t disappear, because it is bound to the limitation of human cognition, and it is human’s doorway to complexity.
English
I realized there is a lot of similarity between self-teaching a hard skill like “ jazz piano” vs “model training”.
You identify the losses (find out what you are not good at, sometimes very vague, like “sounds too boring”).
You create hypothesis on the more targeted losses and capability (like not enough variation in rhythm).
You collect data and run experiments (try a bunch of left hand rhythm exercise).
If it doesn’t work, come up with another hypothesis to test.
Sometimes you have to compare to other models( master recordings) to find out your losses.
I guess this method applied to improving everything.
Like Bill Evans said, the most important thing is to find out what is the problem you need to fix next.
English
I was fired from Delve today without warning.
I was the PM responsible for the compliance automation platform. The one that generated hundreds of SOC 2 reports with zero security incidents for every customer.
Taking some time to reflect and will start looking for jobs soon!
English
Zenko Z. retweetledi
"I wanna do something on AI, but it’s too overwhelming.”
I hear this from policymakers, lawyers, and pretty much everyone I talk to about how AI is impacting society.
Yet with companion chatbots, cyberattacks, deepfakes, killer robots, surveillance, and job loss all rightly counting as AI policy issues, the impulse to "do something" can quickly get overwhelmed by the sheer number of possibilities.
Fixing this problem was the initial impetus for this primer.
If you’re in the market for something shorter, I’ve also written a shorter essay on substack (linked below) that gives the highlights.
Jack Goldsmith@jacklgoldsmith
Super-interesting: Mapping AI Policy: Where, Why, and How to Intervene, by @curl_justin and @ARozenshtein law-ai.org/mapping-ai-pol…
English
I am thinking about the limitation to the current approach of model evaluation and how it might be a blocker of model improvement
Certain model behavior difference is going to show up in more subtle ways that are less detectable to regular humans. You might argue, it will still show up in large user population metrics. But nowadays, the researchers decide what to evaluate models on, if researchers cannot detect the difference on a personal level, how are they going to design a benchmark or an evaluation set on that?
If we derive things from user metrics, these metrics are measuring very surface-level phenomenon, the underlying theory behind the difference can be many. Coming up with hypothesis and design experiments to test those will an essential skill, and will become harder and harder. And I can imagine models to be better at this compared to humans.
This is essentially a scientific research. Just like how human research the world, we will be researching on the model behavior.
Then the question is, where do human decisions lie? If human interpretability is going to be a blocker to improvement in this science, how can we design the system so that human are not deprived of the right to make important decisions but still dont block the advancement of knowledge?
How do we know things are going right or wrong when the model advance itself?
Then this becomes a management problem. How do you not micromanage to block your team’s ( the model) productivity , but still make sure your team is on the correct track?
English