Gary King

New Paper: "Inducing Sustained Creativity and Diversity in Large Language Models" GaryKing.org/quest, with @Queenie__Luo et al. Comments welcome!

Thanks everyone at @JohnsHopkins for coming to my talk & for the fabulous conversations. Great to see all the wonderful changes (with more in progress) in the social sciences there... Slides from my talk on "Who’s to Blame for Survey Instability: Respondents With Random Preferences or Researchers With Flawed Measures": gking.harvard.edu/presentations/…

@SeanLangenfeld You might have a look at this article which explains "why propensity scores should not be used for matching" j.mp/psnot It really is not ambiguous.

I asked chatGPT to explain to me why biostatisticians are so critical of propensity score matching, and I thought the answer was very organized and straightforward. I always summarized it as false confidence, sample bias, and ongoing baseline differences in unmeasured variables. But, I am the farthest thing from being a statistician. Here's the better answer, though: "Great question. Propensity score matching (PSM) is widely used, but many biostatisticians are indeed critical of it—not because it’s useless, but because it’s often misused, misunderstood, or oversold. Here are the main reasons, in a structured way. 1. It cannot fix unmeasured confounding Core limitation: Propensity scores only balance observed covariates. If an important confounder is not measured or is poorly measured, PSM does nothing to fix that. This is especially problematic in retrospective clinical data (e.g., registry, EHR), where: Disease severity, frailty, physician judgment, social factors, etc. are often missing or crude. In surgery and oncology (which you deal with a lot), this is a huge issue: “who gets which treatment” is often driven by factors not captured in the dataset. So PSM can produce beautifully balanced tables and still be badly biased. 2. It can increase bias if done poorly Counterintuitively, matching can make things worse: If the propensity model is: Misspecified Omits important confounders Uses bad functional forms Then matching can: Amplify imbalance in key covariates Induce collider bias Create a sample that is less representative and more biased This is known as bias amplification. 3. It throws away data (often a lot of it) Typical 1:1 matching often: Discards: Many treated patients Many controls Especially if groups are very different initially. Consequences: Lower statistical power Larger standard errors Estimates apply only to a restricted subpopulation (the “matched” patients) So: You trade sample size and generalizability for “balance”. And many papers forget to state that they changed the estimand. 4. People confuse “balance” with “causality” A very common mistake: “The groups are balanced, therefore the effect is causal.” No. Balance on observed variables ≠ exchangeability You can balance 100 variables and still be confounded by 1 unmeasured one. This leads to false confidence in the result. 5. The propensity model is often built incorrectly Common problems: Using stepwise variable selection Using c-statistic to judge PS quality (wrong metric) Including post-treatment variables Omitting weak but important confounders Not checking standardized mean differences properly PS models should: Include all confounders, not just predictors of treatment Optimize covariate balance, not prediction accuracy Many clinical papers get this wrong. 6. It answers a different question than people think After matching, you are no longer estimating: “What is the effect in the whole population?” You are estimating: “What is the effect in the subset of patients who could plausibly have received either treatment?” This is often: Not stated Not understood Not what the clinician thinks they’re reading 7. There are often better methods In many settings, these are statistically superior: Doubly robust methods Inverse probability weighting Targeted maximum likelihood (TMLE) Modern outcome regression with flexible models Causal forests / g-methods These: Use all the data Can achieve better bias–variance tradeoff Are less sensitive to arbitrary matching decisions 8. Matching hides modeling decisions Matching involves many degrees of freedom: Caliper size With or without replacement Ratio (1:1, 1:2, 1:5…) Which variables go in PS How ties are handled Each choice: Can change the result Is rarely pre-specified Enables subtle p-hacking 9. In medical literature, it is often used as a “ritual” Unfortunately: PSM has become a checkbox: “Do PSM → claim causal inference”. Many statisticians view this as methodological cargo cult science. The fair, balanced conclusion Propensity scores are: ✅ Useful when done carefully ✅ Better than crude regression in some settings ❌ Not a magic fix for confounding ❌ Often misapplied in clinical research ❌ Frequently over-interpreted A practical rule of thumb If: Treatment choice is strongly influenced by unrecorded clinical judgment Or frailty, anatomy, tumor resectability, performance status are not well captured Then: PSM probably does not rescue the study from confounding by indication."

New software: "Projoint: The One Stop Conjoint Stop" (with @aaronrkaufman & @YusakuHoriuchi), makes conjoint surveys easier & less biased, including everything from a drag-and-drop web survey design tool to specialized analysis software. Comments welcome. yhoriuchi.github.io/projoint/

New paper, with @mitsurumu: "Survey Estimates of Wartime Mortality". Comments welcome! GaryKing.org/sibs

New paper:"Experimental Evidence on the (Limited) Influence of Reputable Media Outlets" w/Bharat Anand, Kiran Misra, Sascha Riaz at GaryKing.org/reputable

Big congratulations to Chris Kenny, Harvard's newest PhD, with a few of his fans.

My talk on measurement error in conjoint survey analysis at UCF. youtu.be/yVuiAh59Hew using slides (download and view in Adobe Reader or Skim): gking.harvard.edu/presentations/…

Slides for my talk at UCF on "Correcting Measurement Error Bias in Conjoint Survey Experiments" (now forthcoming in the AJPS; GaryKing.org/conjointE) along with work on Survey Instability: gking.harvard.edu/presentations/… Thanks to everyone at UCF for a great visit!

Super excited to announce the #PerusallExchange 2024 keynote speakers @stephen_wolfram (CEO/founder Wolfram Research) and @MIT Prof. Jesse Thaler (Director, Institute for AI & Fundamental Interactions). Don't miss this! register hubs.li/Q02vyhqw0 #AI #teaching

Slides for my 3/13 Caltech talk about "Is Survey Instability Due to Respondents who Don't Understand Politics or Researchers Who Don't Understand Respondents" based on a paper-to-be with Libby Jenke gking.harvard.edu/presentations/… Venue: hss.caltech.edu/news-and-event…

Slides for my 3/12 UCLA talk about "How American Politics Ensures Electoral Accountability in Congress" based on paper w/ @Jonathan_N_Katz & @DannyCEbanks gking.harvard.edu/presentations/…

#Dataverse2024 @kinggary won't physically join us this year, but in 2017, everyone was trying to catch those marshmallows! #10thannualmeeting The 2024 meeting starts on 3/4/24 and Zoom access and registration will be available until Monday at 3pm. sites.harvard.edu/dcm2024/agenda/

@rmkubinec @namalhotra @carlislerainey @matt_motta Our behavioral models (our observation mechanisms) can be tremendously important. Economists originally liked it because they can still get (stochastic) rationality, but only by assuming humans have no stable preferences. much evidence rejects this model

@namalhotra @carlislerainey @matt_motta We had a conversation about this at polmeth mena. Authors have considered random utility models but don’t want to commit to a behavioral model at present.

Intuitive and simple solution to a common problem. I really like this approach!

@namalhotra @matt_motta Thanks the comment, Neil. The random utility model is an assumption (humans have random preferences & never make mistakes in survey responses) not supported by the evidence. This often doesn't matter but does here. Have a look at our supplementary appendix, which discusses this.

@matt_motta I may just be really dumb but I don't understand this paper. Normally, one would write out a random utility choice model (e.g, Luce/Thurstone), which have error terms. But that doesn't mean you inflate up estimates as if it was a compliance problem.

Slides for my Friday talk at the Harvard Experimental Working Group on "Correcting Measurement Error Bias in Conjoint Survey Experiments" gking.harvard.edu/presentations/… based on GaryKing.org/conjointE w/ @katie_clayton14, @YusakuHoriuchi, @aaronrkaufman, @MayyaKomis

Slides for my talk at Harvard Law School tomorrow on "How to Measure Legislative District Compactness If You Only Know it When You See it" gking.harvard.edu/presentations/…

@kinggary @JohnHolbein1 Plus way better in the top quartile of journals (74% in 2022)! Still plenty of room for improvement to be sure, but not all bad news here.

@carlislerainey @JohnHolbein1 Definitely right. As social scientists we know it is hard to change individual behavior, even when the individuals are us! Nice to see that expectations have changed so clearly, and that behavior is following.

@kinggary @JohnHolbein1 I think there's a generational divide in how one interprets these results. Post ~2014, sharing is expected, common, and easy, so it's "only" 31% On the other hand, what a massive change in 30 years.

Oof. This is a gut punch. "We find that relatively few [political science] articles make the underlying data & code available to others" "Reproduction archives are almost entirely unavailable from 1995-2006" In 2022, 31% of quantitative PS journals include shared data & code