Karl T. Muth 🌐✈️📊

After two decades of research into digital libraries and ML/AI models, I’m thrilled my research on #AI #ModelCollapse is published today in IP & Comp. L.J. I use tools from economics to examine the problem. My thesis? The digital economy is facing a new "Great Inflation." 🧵👇

@k1290c @neerajadeshp @MariGO2thepolls If you don't understand why comparability matters when looking at two cases, then we probably shouldn't continue a conversation. Enjoy your evening.

I don't understand your inclusion of the poverty rates if they didn't mean anything to the larger point. It's kind of a silly comparison at best. We don't have to get into the weeds to understand that MS is mostly a shithole and Chicago is one of America's top cities. A few points here and there in MS' favor doesn't really change that reality.

Bad governance outside of housing stuff in blue states is greatly exaggerated tbh. Do I wish they were governed better? Sure. A lot to be desired but it’s all awash compared to housing. If is were truly badly covered horsing would be cheap cuz it wouldn’t be where everyone wants to live

@k1290c @neerajadeshp @MariGO2thepolls No one claimed ten years of improving childrens' education would move the poverty rate significantly; not only do I not claim that, I think it would be absurd to claim that. My point is the similarity of these populations on two vectors.

@KarlMuth @neerajadeshp @MariGO2thepolls Looks like basically the same poverty rate? Also comparing a city to a state is questionable at best.

@CalumDouglas1 You cannot read Parkinson, Orwell, etc. and not marvel not only at the quality of writing civil servants and bureaucrats with unremarkable job titles were capable of, but also wonder what they would've done if given even a tiny bit more agency or amplification.

I think on a personal level, the most interesting and also depressing aspect of very detailed study of WW2 administration of defence and industry in Britain, is seeing how exceptionally competent almost all of Britains administrators were. You almost get cognitive dissonance just reading half the files, trying to work out how its even the same country we live in now which once produced these kinds of reports. There is no doubt there has been a progressive, and disastrous collapse in the all round general collective intellectual capability of the civil service and defence administration in Britain over the last few decades. Its possible to argue this is inevitable after the system was optimised by the white heat of war, and the dead wood was scattered to the four winds by virtue of necessity, but it doesnt change the fact that its almost impossible to reconcile the standards which were once taken for granted as a matter of national survival, with those we see today. You can see it at every level, and even in my small town, talking to retired councillors, they cant believe the desperately poor standard of those currently doing the jobs they did 30 years ago. How do you keep the best of your systems intact passing from wartime to peacetime ? Has anyone solved this question ? Perhaps, as far as I can see from a brief search (this is not my specific area of historical study) Singapore is one example of the most valiant attempt, with some measure of sucess. This has been discussed in "Meritocracy and the Singapore Political System." Asian Journal of Political Science. (link in comments), which describes the strict measures taken post independance in Singapore to introduce performance based merit in the Civil Service, and intensely rigid anti corruption laws. Letter below from 21st November 1935, Defence Requirements Sub-Comittee of the Committee of Imperial Defence - the CID, (Chamberlain presiding) A year before, on the 8th October 1934, Chamberlain had been lambasted by Lord Hankey, for expanding the RAF by ten squadrons over and above that even recommended by the Defence Requirements committee. Chamberlain suceeded in his push to expand the RAF at home as rapidly as possible. These meetings were however, all secret, and were not declassified until the 1970`s. The push for re-armament was not fully revealed to Germany, because the CID had agreed in 1934, that it would need five years to prepare for war with Germany, and that every diplomatic measure possible was to be taken until that date (1939) to avoid the outbreak of war with Germany. It was then, after Chamberlains withdrawl from politics and death, taken as the established narrative that Britain had NOT begun large scale and direct preparations to defeat Germany before the beginning of Churchills tenure. Only after the Committee of Imperial Defence files were declassified, covering what was really happening in British war planning in the 1930`s, that the truth became apparent. The established story of British stupidly and appeasement of totalitarianism before Churchill was Prime Minister, were utter nonsense - but, had been important to maintain the illusion of until 1939.

@k1290c @neerajadeshp @MariGO2thepolls Mississippi is a true success story for statewide coordination meaningfully impacting median student outcomes. Mississippi (38% Black, 18% below poverty line) Chicago (29% Black, 17% below poverty line) Chicago has no excuse for its perennial sabotage of young folks' futures.

@neerajadeshp @MariGO2thepolls On what basis do you say that?

I don't often comment on car design, but the new @Ferrari is adequately atrocious to attract comment. Am I the only person who saw this car's proportions and IMMEDIATELY thought, "you can't fool me, bro, that's a Honda CR-X profile on some Aerodiscs..." x.com/i/status/20590…

@AndrewDamitio I do not often say things like this, but you really need to engage with the development literature more comprehensively. These arguments are not only considered but exhaustively explored. It's been a long time since Popkin's The Rational Peasant and various subsequent rebuttals.

In future decades economists will write about an "upper income trap," wherein most wealthy economies stop growing due to bureaucratic dysfunction, service sector cost disease, social unrest from a highly educated populace, rent seeking, and anti-industrial sentiment.

Thanks for the thoughtful reply: I really appreciate you engaging on this. I don't disagree with your overall position at all. For core/bar courses especially, the looming exam as a strong incentive to outline, study, and master the material is often highly effective at driving (a) and (b) in your framework. In this particular elective (non-bar), I'm treating the exam more purely as a measurement device rather than embodying the incentive structure. Students in these elective classes (just my opinion, your mileage may vary as they say in the automobile advertisements) tend to bring more intrinsic motivation and fundamental interest, so the risk of someone coasting entirely on the appendices feels lower (and less pedagogically disastrous) than it would in, say, Torts or Income Tax. This exam design is an experiment in seeing whether we can get closer to measuring the "value added" skill that will matter in practice while still producing a meaningful grading distribution that distinguishes lesser fluency from deeper understanding. On a personal note, I came from high-stakes memorization-centric testing environments (whether UChicago's "but did you remember this nuance..." flavor or the London School of Economics's notorious comprehensive exams, which made winter and summer breaks feel paid in full) and I think I'm better for them and have benefited from the incentive you describe. So I do believe those are the right environmental aspects for an exam to embrace in many settings and contexts. We'll see how things evolve and what changes (or doesn't) as the world changes around us. There are young lawyers today who've never really had to walk the stacks after dinner and pull cases, who don't really understand viscerally that volume-reporter-page in a citation really is a volume on a shelf in the law library, and the profession is worse for that modernization (so I am not without nostalgia, trust me). To them, a citation is like a website's address, just an arbitrary location marker of where the data they need lives. I favor some mix of old and new and we'll have to experiment to get there. But the training in our discipline has moved from pencils to word processors in the period during which medical schools have moved from hacksaws to lasers, so we have the luxury of moving deliberately and I've no doubt we collectively, if not individually in each case, will do precisely that.

I think your argument hinges on a particular vision of the purpose and of exams that, with respect, I just disagree with. If I follow you correctly, your view is that an exam should recreate a realistic practice situation and test to see how much a student adds value. But I don't think that's what exams are supposed to do at all. As I see it, exams basically do two things: (a) create the incentive to learn material, so that the student knows material, and (b) provides a test of relative fluency with complex legal materials, and especially the student's ability with the core legal task of identifying the issues that a client would have and how they could be addressed, so that prospective employers can compare a potential hire's relative degree of legal reasoning skills.

I know there are many (understatement) approaches to AI use where students are being evaluated, and that there is variation between disciplines and levels of study, but I thought I'd share one and perhaps stoke the debate. Anyone (including my students) is welcome to comment...

Aspects of what you describe seem not unlike the Socratic environment of the law school classroom. Where the two (quickly) diverge, however, is the introspection required for one and not the other. A system (whether an LLM or a flip of a coin or a Ouija board) cannot feel or make internal moral judgments on behalf of its operator. In fact, offloading or delegating these judgments invalidates them. I agree. If part of what you are teaching is introspection or an exploration of what a person sees as righteousness or piety, these are personal matters that are inherently linked to individualism, not external processes. Meanwhile, law is not (at least in my classroom) an Aristotelian simulation of what justice might be in the abstract or a Rawlsian puzzle of imagining the injustice suffered by another person. Rather, it is a mechanical examination of how rules are applied to facts and how standards are either met or not met. In this sense, it is entirely external and indifferent to the operator. If I tell you that burglary is the 1) breaking and 2) entering of the 3) dwelling 4) of another 5) at night with 6) the intent to 7) commit a felony therein, I do not care about your opinion of what a burglary is. I've provided you the definition of burglary. You can do two things with this definition; one is theory and one is practice. Theory I do care, however, and care deeply, if you can look at a pattern of facts and discern whether it is plausible that each element (remember that all elements must be met) is met under the conditions as stated. Practice I also care, further, if you can think like a prosecutor (or like defense counsel) and examine the strengths and weaknesses of various arguments that might affect the applicability of the definition to a scenario. In my research on LLMs where we give them limited (or purposely sabotaged) libraries to draw upon or give them adversarial scenarios (like serving on a jury with other LLMs likely to disagree), this is informative. But it is informative in the sense that studying how a set of gears works is informative; it is not informative in the sense that introspective or philosophical or theological inquiry is informative. It offers no such value. The closest thing in my law school classroom to your concern is that a student can ask the LLM for what she thinks she wants. But she cannot ask the LLM for what she should want or could want or wants to want. When the man walks into the pet store and asks for a carpet cleaner good for dog urine, he may instead need a leash. Or a manual on better dog training. Or to leave work earlier and have more compassion for his dog. One can be a person who wants French fries. But that same person can be a person who wants to want a salad. And those two things can coexist in a person. And your work delves into this very human complexity. I just care whether a person is a burglar and I have a handy checklist to tell whether they are or not. Whether I use that checklist or my student uses that checklist or a robot uses the checklist, they're a burglar. Or not.

@KarlMuth This is also why creating atmospheres of fear and stress (as noted in my comments about the OP being cop-like) are detrimental to the educational process as well. Ignatian pedagogy functions best when the students are as present as possible, with as little fear as possible.

As a professor, I would not do this. I hate it when my colleagues do this. I also understand why they do this. I do not use LLMs in my work. I hope my students do not use LLMs in their work. But I absolutely refuse to become a cop to try and control my colleagues or my students.

@DaultRadio You certainly don't need my permission to wander in a different direction and there are plenty to choose from. I'm curious (as someone whose research is very much focused on various machine learning models including LLMs) how you arrived at your views on their pedagogical use...

@KarlMuth [nods] I did read your post. I am taking the subject in a different direction, with a different emphasis. I hope that’s okay with you. I also hope you rethink your use of LLMs in the classroom, but that’s a different conversation.

@beegboulder @OrinKerr Anticipating and addressing your question is the purpose of this sentence.

@KarlMuth @OrinKerr But you’re affirmatively saying that the appendices would receive a certain grade. You don’t make clear that the grade you say they’ll receive is subject to reduction based on the curve.

And I didn't mean to seem curt here, Orin; I concede a student *could* rationally coast on Appendix A for a B+/B (or a B- or C+ if many peers substantially outperform the AI answers, pushing them further down in the grading stack) and redirect effort elsewhere during the term (work, online dating, videogames, who knows). That's a real risk and I'm not dismissive/ignorant of it, but I don't see it as a fatal flaw, either. Nor do I see it as unique to AI (if I provide an example of "this answer got a C last year, so you'd better do at least this well and hopefully better" with a human-written essay, same result). That said, I see it as a feature in some ways: law practice (clerkships, BigLaw memos, client advice) is full of imperfect baselines from colleagues or (nowadays) AI. The skill I'm trying to build is *adding value* beyond the baseline, not just regurgitating it. Students who phone it in will get exactly the grade the curve assigns to "baseline + minimal polish" and future employers/judges will notice the difference in their work product. I'm not claiming this is perfect (no exam design is), but it feels closer to preparing them for the actual environment they'll face than pretending perfect answers appear in a vacuum or that law is the one sacred corner of the professional universe where AI doesn't exist (hence my comments elsewhere that Berkeley may think their hardcore AI ban will lead to their students' writing being the fine Amish furniture of legal prose but I doubt they'll get the result they're looking for).

@OrinKerr Sure. And, similarly, they can go do a clerkship or a law job (or a non-law job, we produce a non-negligible number of consultants, lobbyists, etc.) where they just hand in what they get from an LLM and add little or no value to the baseline answer. But high performers won't.

@PeterMoskos Okay, different argument: 🚑 outcomes for GSW victims are not much better w/ ShotSpotter or similar tech than without 🚓 cities are not set up to respond quickly to events and see these calls mostly as a drain on resources 🔫 normal policing activity locates more illegal guns.

I have repeatedly (and politely) tried to engage with any opponent of gun shot detection who talks of "tons of false positives" (often citing a NYC Comptroller Brad Lander report) to simply define what they think a "false positive" is. Crickets. There's something else going on.

@japan_nobunaga I've written extensively about this. But it is something far left politicians, especially mayors of cities, often try to limit or destroy. It is always on the mind because we must be so constantly vigilant to protect these rights from these politicians... journals.law.harvard.edu/blackletter/wp…

@KarlMuth その「素晴らしい自由」というのは、例えばどんな時に感じられるものなのでしょうか？

Hello from Japan 🇯🇵👋 I have a question for Americans 🇺🇸🙋 I've never seen a real gun in my life. Only in anime and movies. Is it true that Americans sometimes see guns at stores like Walmart? In Japan, we only see them on TV. Is that normal in America? 🤔

@GhostOfSocrates Yes, I'm sure it's unimaginable that guy who posts exam cover page provocation who teaches at elite law school might be talking about that context and not people studying SCI-0100 "Rocks and Stars and Stuff" at the local community college.

"this is not what you're imagining." rofl. the guy posted with zero context and now we're supposed to imagine that he was referring specifically to a narrow use case that is like 0.001% of the educational context for students in America?

@GhostOfSocrates Cool, I teach zero of those people. This is an elite law school. This is not what you're imagining.

@KarlMuth That applies to a narrow subset of people graduating from university. For the vast majority it doesn't matter what their grades are for a particular class. If you're in that subset then sure optimize for top grades.

My advice for most students would be to just use the LLM answers. C's get degrees.

@Major_Ferret @ClappDescendant Markets run on information. Give everybody an A (and frankly everybody at @NorthwesternLaw could be an A student at [pick a school]), we deprive the market of that incremental information. I'm cited by the SEC 20+ times in Final Rules. The common theme? Markets need information.

@Major_Ferret @ClappDescendant Great question. Certain firms (I'm looking at you, Jenner and Winston) care more about grades because that's their culture and they've had luck filtering on that basis. Certain judges care a lot about law review + GPA for clerks. Others care less. It is, and should be, a market.