scott cunningham

I’ve been doing a series on Claude code for “quantitative social scientists” on my substack. It’s not much a series I guess as much as I am regularly documenting what I’m learning. It started mid Dec 2025 and will continue. causalinf.substack.com/s/claude-code

Hayek’s classic insight was that much economically relevant knowledge is dispersed, local, and often tacit. That is one reason decentralized markets have historically outperformed central planning. But AI can change those “knowledge physics” by making more of that knowledge codifiable, transferable, and usable at scale.

The @nytimes piece today by @ByrneEdsal13590 highlights a concern I share: “If we stay on the current path, the risk of extreme concentration — both economic and political — is very real.” In work with @zhitzig, we ask why AI may shift the balance between dispersed knowledge and centralized control.

I wrote a substack today where I tried to explain my views about how complementary Claude code is to your work when you have real expertise in that specific area, and how sketchy it is when you don’t. causalinf.substack.com/p/claude-code-…

@drStuartGilmour @ashwinmalshe I’d be curious to hear what you find

@drStuartGilmour @ashwinmalshe You should do that. Then run it across all six packages in python, R and Stata. Create massive imbalance on ANY variable with this level of scale. Let it be 10^10 and then add in more covariates. Tell me what the package variation is. Then show what happens under many Xs

Some of the people commenting negatively on this were engaging positively with Cunningham’s package replication that makes the same classic error, just yesterday. This field has no concept of how to handle count data!

@drStuartGilmour @ashwinmalshe Just ran areg. The coefficient on poptotaltrend is zero and insignivant. The variable does nothing in plain regression. The r^2 is 0.44 coming basically from the municipality fixed effects. The problem I identified wasn’t a conceptual relationship bw homicide rates and pop.

@causalinf @ashwinmalshe It’s worth noting that even areg can’t handle that dataset. I’m not sure you can draw too many lessons about the packages from such a bad model

@andy_garin @PhillipsEcon I hear you.

@causalinf @PhillipsEcon Noted and appreciated. Just wanted to be clear that while I tend towards skepticism I am *not* trying to blindly dismiss these approaches wholesale.

Results from my Claude Code audit of six Callaway and Sant'Anna packages (two in python, two in R, two in Stata). Same specification, same dataset, same covariates, same estimator, almost never do they agree. open.substack.com/pub/causalinf/…

@andy_garin @PhillipsEcon Yeah, I entirely missed all these. Sorry. I’m so out of the practice of reading these Twitter threads; it’s been three years since I’ve been on here and I forgot how I used to do it. I also get defensive on here i am realizing.

@causalinf @PhillipsEcon If there’s a specific reason to believe matching on certain controls satisfies a CIA or conditional PT assumption, that’s great an justifies these approaches. But I know very few people who are convinced by “I matched on some controls I have so this is better”

@andy_garin @PhillipsEcon I probably can’t follow the convo tbh on twitter. I get lost in these threaded convos so if I am not following that’s on me

@causalinf @PhillipsEcon Sure, but this is very clearly not what I said. You don’t have to agree, but I want to be clear you are not correctly representing my point.

@ashwinmalshe @drStuartGilmour It’s definitely given me pause. We often master the econometrics, the dataset, and the syntax of our preferred language but not necessarily the underlying source code of the packages. The estimates *should* be identical across all packages and the variation was not trivial at all

@causalinf @drStuartGilmour Btw I saw your post on LI and thought it was really eye-opening. When I switched to R more than a decade ago I used to compare Stata and R results. It was a real PITA but now with AI this is so convenient

This is important from @causalinf: Six Callaway and Sant'Anna DiD packages produce different results once covariates are introduced. "Package choice is a substantive decision that determines whether your ATT is 0.00, 0.45, 1.15, or 2.38..." substacktools.com/sharex/k15_2gyd

@drStuartGilmour How does that explain the variation in packages? They are *all* using the same LHS, same RHS, same estimator. Fyi, there was no “right hand side” variables. Refamiliarize yourself with CS.

@causalinf You put a rate on the lhs and then the population used to calculate the rate in the rhs. Also because the rate is small many of your predicted values and CIs would be negative. I know it’s not your paper but that’s bad practice.

@andy_garin @PhillipsEcon To say you know ppl who say if unconditional PT is implausible and cannot therefore hold for CPT, and therefore they refuse to use did, well honestly that’s not my business. You shouldn’t use a research design you think is wrong, regardless. 3/n

@andy_garin @PhillipsEcon But if you’re saying you don’t believe that parallel trends can hold for CPT but not PT, I don’t know what to say bc that’s incorrect. It absolutely can. If you’re saying on vibes, you aren’t persuaded than that’s fine. But it’s trivial to give examples of this. 2/n

@notcalledjack 5 out of 5 Jack!

@causalinf don't do this to me

Highly recommend. 5 out of 5 on the peanut butter chocolate scoring system I use. I’ve never given a 5 before in my life too!

@ashwinmalshe @drStuartGilmour Mine like you said was the same estimator and same spec for six packages — so anything in one should’ve been in the others.

@ashwinmalshe @drStuartGilmour Oh mine was definitely not misspecified.

Dear followers, I’m happy to share this new academic paper on how even capable AI can lead to deterioration of collective knowledge in society