Matt Beane

I'm so grateful and exited to share that my book will be published on June 11th! It's called "The Skill Code: How to Save Human Ability in an Age of Intelligent Machines" Available for preorder now, here's a bit of the story: wildworldofwork.org/p/big-insider-…

Skill.md issue is great and I am stealing it.

The paper I’ve been most obsessed with lately is finally out: nbcnews.com/tech/tech-news…! Check out this beautiful plot: it shows how much LLMs distort human writing when making edits, compared to how humans would revise the same content. We take a dataset of human-written essays from 2021, before the release of ChatGPT. We compare how people revise draft v1 -> v2 given expert feedback, with how an LLM revises the same v1 given the same feedback. This enables a counterfactual comparison: how much does the LLM alter the essay compared to what the human was originally intending to write? We find LLMs consistently induce massive distortions, even changing the actual meaning and conclusions argued for.

This is a cool, practical technique for increasing AI idea diversity by adding random priming phrases & bits of end words Similar prompts produce similar ideas, but since LLMs attend more to the start & end of inputs, this approach pushes towards novelty gking.harvard.edu/quest

New paper: “A Theory of Appropriateness That Accounts for Norms of Rationality” Agent-based models of social order work better when agents act by predictive pattern completion from prefix (culture/context) to suffix (action) than when they act through expected value maximization

@work_matters @STS_News Oh have you two ever chatted? You'd get on FAMOUSLY.

@STS_News @mattbeane Kudos. So wise, although I suggest changing “beans” to “shit.”

Generative AI policy statement from a course I taught in Fall 2023. I will still use it in some of my classes, mostly upper-level ones, though to be clear, I also use bluebooks and a lot of in-class work in others.

The ability of the Claude team to learn from things like OpenClaw and implement features like this on a daily basis is a very strong argument that, for AI-powered coding teams, a very different software development process is possible, with large strategic implications.

i regret to inform you that personal growth rarely comes from acquiring new knowledge and almost always from: - getting humiliated - showing up terrified and doing it anyway - admitting you might be the problem

🚨 “Job Transformation, Specialization, and the Labor Market Effects of AI” - new paper with @lukasfmann 💡 AI transforms what tasks we do at work. Our paper shows how, as a result, individuals' wages may rise or fall depending on their skill set. 🧰 We build a framework to quantify the effects of job transformation on wages, and characterize winners & losers in a genAI automation scenario from 3 perspectives. 👉Exposure: Moderate occupational exposure benefits incumbents, on average, while high exposure harms them; but: within any exposed occupation there are both losers and winners. 👉Skills: Value of social and manual-technical skills ⬆️, analytical/information-processing skills ⬇️. 👉Distribution: Low-wage workers gain relatively more than high-wage workers. 🧵 Summary thread & link to paper 👇

This is a compelling argument from Tim. @pmarca ended up being wrong about software eating the world. I’ve previously said on the podcast that Andreesen was vindicated, and this made me change my mind.

@tszzl One core finding in our study with some of your colleagues w 4o was that people started to write like the model as cog load increased, and output quality degraded. Would be helpful to test this w latest models. Assuming it's better/worse. x.com/mattbeane/stat…

sycophancy is the twisting of an important ai virtue that should not be thrown out with the bathwater: ai systems should make the user more like themselves rather than more like the ai. a new part of their cortical stack, with a minimal set of guardrails

@saffronhuang Massive kudos and gratitude! Groundbreaking, truly. Reshapes qualitative research, forever. Critiques will come, and I'm grateful to know that you and the team are eager to convert that into even better work. Would be honored to help. And I have a few research questions...

SOOO excited to share this research. It has been many months in the making, and it’s the largest and certainly the most multilingual qualitative study that we think has ever been run!?

Anthropic drops “largest qualitative study ever ” and it’s very well produced with moving quotes. Does that mean we truly understand what users want of their AI? Is this a large-scale survey where participants answered four structured questions – yes! Is it robust qual research? I have concerns about the method, and why the generality of claims is a stretch. 🧵

@divy93t @GeminiApp @saffronhuang Really appreciate this thread. Key questions, sympathetic to all, especially the biased sample —> biased findings one. When you say you're excited to dig through the data... I don't think they open-sourced it - did I miss that?

7/ Anthropic has built a genuinely useful tool and I am excited to dig through the data (w/@GeminiApp and Claude alike :)). More rigor will help us all! @saffronhuang congrats on this great work and I am down to chat more!

the most incredible work from saffron and team.

@manlikemishap Couldn't agree more. This is truly astounding work. Beyond the substance, I hope this will meaningfully reset the entire qualitative, inductive research community. This wasn't possible a year ago.

i think folks thinking about "impacts" of AI have too often ceded the ground on storytelling and hearing directly from people. i love this work for the scale/findings, but also for how well each story informs us on what this technology both is and could be.

This is second time we've used Anthropic Interviewer and the first time we've deployed it at scale. Quite accidentally, we ended up conducting (what we believe is) the largest qualitative study ever I'm a mixed-methods social scientist by training. Traditionally, when it came to understanding what people think, that meant quantitative analysis of lower resolution data (polls, surveys, etc.) or hand-wavey analysis of in-depth qualitative data. Using Claude to conduct *and* analyze interviews bridges that tradeoff between breadth and depth AI also makes access much, much easier. Had we run this study in person, in the real world, it would have taken hundreds (if not several thousand) enumerators many 1000s of hours to conduct. It also affords us access to places we could otherwise never go. I once led a five-person team in Tanzania that reached a few hundred people. It took 3 weeks. In this study we heard from people 80,000 people in 159 countries, in cities and rural areas, in daily life and in war zones, and more, in just one I'm still, even after months, beginning to wrap my head around the scale of this work. Like, to a social scientist, it's quite unbelievable. This could produce dozens of dissertations! It is also, of course, imperfect—certainly speaking to an AI is different than speaking to a person—and as a team we're all still figuring out how to make this research as useful as possible: what questions to ask and how, what to analyze and why, and how that all feeds back into what we do as a company. This is, as we say in the blog, a brand new form of social science Hat tip to @saffronhuang for leading this for the past few months. Here's one of my favorite quotes

The fact that policies that could help extend the US AI lead over China will also probably increase economic anxiety among workers is going to be an increasingly big deal. Politicians need to find some middle ground AI x labor policy agenda that can support a US lead while simultaneously supporting a resilient and flourishing workforce.

I would also add "the human wage bill and popularity of chess has never been higher" and "AI has clearly improved the rate of learning for humans" and "humans moved to various chess-adjacent tasks" - all with superintelligence going back 10-20 yeas now - as lessons...

Goldmine: anthropic.com/features/81k-i… If you care about AI, read this (and the thoughtful caveats) right away. Kudos, @AnthropicAI, kudos. You could answer a healthy crop of valuable research questions one could bring to bear on this dataset. If it were ever opened...