Aryeh L. Englander

@tyler_m_john Yes, but it's also just a straightforward consequence of risk aversion.

@AryehEnglander I believe risk aversion for HO credences is described as "ambiguity aversion" in the philosophical literature globalprioritiesinstitute.org/wp-content/upl…

OK since p(doom) is discourse here is my view on communicating risk with probabilities. We should do it because it makes it much clearer to people what you think and is empirically demonstrated good epistemic practice. Also we should hedge to show high-order uncertainty. Some theses: 1. Probabilities give people more insight into what you think. If you use vague, qualitative language instead of numbers, people will just assume what you mean. There's @PTetlock's famous Bay of Pigs anecdote, where an advisor told Kennedy there was a “fair chance,” meaning a 25% chance of success. Kennedy later reported he had assumed the advisor meant a 75% chance, and said he wouldn't have pursued the invasion if he had known the advisor only meant 25%! But this kind of miscommunication is ubiquitous. People assume different things about likelihood when speakers use qualitative language — it's an inherently less clear way to communicate what you are thinking. If you want your speaker to understand you, use numbers! Or at the very least, refer to the literature on perceptions of probability (see below) and pick your qualitative term very carefully so you communicate the right range! And don't use the extremely vague terms like "fair chance" or "improbable" that could mean literally anything to your listener. That is an extreme form of carelessness that we don't criticize often enough. 2. There haven't been many clear findings from the science of forecasting, but one of the clearest findings is that you make better predictions when you use precise numbers, even if these are completely made up. This is also true in group settings when aggregating the judgments of many people — which is essentially an idealized version of what we're doing pretty much any time we talk about probabilities. academic.oup.com/isq/article-ab… Here is an old thread I wrote on this topic some years ago: x.com/tyler_m_john/s… 3. Yes, people do perceive numbers as signaling more authority, and we shouldn't signal more authority than is appropriate. (How much is appropriate? Depends on the context. There isn't a universal answer in the context of existential risk from AI.) But you can do that without dropping numbers and losing the benefits of numbers I just set out. For example you can just use couching language, like "I would guess roughly 20%, but huge error bars, no one knows." 4. This can be studied!! It has already been studied a lot. I am finding it frustrating that no one in this debate is citing actual literature on perceptions of probabilities, especially in the age of LLMs where this information is readily available. We do know that percentages are viewed as more credible than qualitative language: papers.ssrn.com/sol3/papers.cf…. We do also know that hearing "61.87%" rather than "60%" triggers the inference that the speaker must have epistemic access that warrants the extra digits. frontiersin.org/journals/psych… How much higher-order confidence is it appropriate to convey when communicating the P(doom) of, say, a world expert on AI or an aggregate survey of every AI researcher publishing in NeurIPS? I don't know! If you want to make an argument that saying "20%" signals too much confidence, please cite some of this literature and explain why you think that the groundedness signaled to the audience is inappropriate. If you do want to advocate for a different communication style, it is not expensive to run a quick MTurk study to see what people's perceptions of it are and compare it to default rhetoric. Or even more cheaply you can run it on LLMs, which are a decent natural laboratory for testing hypotheses about human psychology in the absence of humans to test on. I hope to practice what I preach in the coming days and run some more LLM tests (I've ran one N = 7000 test yesterday) and set up a Mechanical Turk account so I can test my above claim about couching probabilities being just as good as using qualitative language, but with more clarity in communication and better epistemic practice.

@davidmanheim @tyler_m_john @yudapearl Can you elaborate?

@AryehEnglander @tyler_m_john @yudapearl Yes, joint probability distributions are complicated, and splitting them as conditionals over causal graphs allows you to represent it both causally and more compactly. Alternatives, though, are mostly not amenable to coherent updating, much less correct for causal reasoning.

@davidmanheim @tyler_m_john @yudapearl IIRC he replaced them with a long list of conditional probabilities. In some cases that way is easier, in others it seems far more unwieldy.

@AryehEnglander @tyler_m_john @yudapearl explained why these higher order probabilities aren't needed in a older paper, "Do we need higher-order probabilities and, if so, what do they mean?" (Not sure if there's a copy online, I got a physical photocopy in grad school when working on this.)

@slatestarcodex @deanwball @MInusGix @JeffLadish @Raemon777 @ESYudkowsky You missed the opportunity to do this in the style of the first Mishna of Pirkei Avot.

What? You dare accuse us of not being situated within a lineage?! Our lineage is top notch! Most of us were initiated by @Raemon777 at one of his secret solstices. @Raemon777 was initiated by @ESYudkowsky at Benton-That-Was. @ESYudkowsky was initiated in an epileptic vision, St-Paul-style, by ET Jaynes. ET Jaynes was initiated at Cambridge by Harold Jeffreys. Harold Jeffreys was initiated in a storm off the Cape of Good Hope by Francis Galton. Francis Galton was initiated by Jeremy Bentham's Auto-Icon (it still counts! he formed the intention before death, and his body was almost perfectly preserved!) at University College London. Jeremy Bentham was initiated by Pierre-Simon Laplace in the Paris Catacombs. Pierre-Simon Laplace was initiated by The Reverend Thomas Bayes in Tunbridge Wells. The Reverend Thomas Bayes was initiated by either Isaac Newton or Gottfried Wilhelm Leibnitz, apparently the records are unclear and fiercely contested, but it doesn't matter because both of them were initiated in the same Amsterdam lens-grinding workshop on the same day by Christian Huygens. Christian Huygens was initiated by Rene Descartes, who fools and unbelievers claim "founded" the Rationalists, on the point (1, 1) on the coordinate plane. Rene Descartes was initiated by Francis Bacon during a secret meeting in Calais. Before Francis, the lineage apparently stayed in the Bacon family for seven generations, all the way back to Roger Bacon, whose brazen automaton was an early, though abortive, attempt at pre-empting bad actors and building a safe superintelligence. He was initiated by Leonardo Fibonnaci, who was initiated in Morocco by Nizami Aruzi, who was initiated in Nishapur by Omar Khayyam (the regular one, not Ravenhurst). Khayyam was initiated by a nameless philosopher who fled Baghdad during the persecutions of al-Ghazali, who was initiated by ibn al-Haytham, who was initiated by ibn Qurra (there may have been several ibn Qurras, the record is unclear), who was initiated by al-Khwarizmi (of "algorithm" fame!), who was initiated by Alhazred (falsely accused of being an early 20th century invention, but we know the truth), who was initiated by ibn Hayyan, by John of Damascus, by Jacob of Edessa, by Stephanus of Alexandria, and then something like thirty different Alexandrians I can never keep straight, mostly corresponding to the Secret Keepers of the Great Library. Origen was actually one of ours (the part with everyone being perfect spheres in Heaven gives it away), as was Heron (who aborted his incipient Industrial Revolution after realizing that it would increase existential risk too much if we got ultratechnology before Christianity had finished pacifying the West), as was Demetrius of Phalerum. Demetrius was initiated by Epicurus (so far nobody has noticed the sacred geometric similarities between Lighthaven and the Κήπος of Epicurus, probably because nobody except us retains the map of the latter). Epicurus was initiated by Nausiphanes, Nausiphanes by Diogenes of Smyrna (NOT the barrel-dweller, different guy with the same name!), and Diogenes by Democritus (Scott Aaronson's "Quantum Computing Since Democritus" is a nod to some of the content in their secret dialogues). Democritus was initiated by Leucippus in Miletus, Leucippus was initiated by Aristaeus in Croton, and Aristaeus was initiated by Pythagoras, who personally invented Reason in the year 536 BC after staring too hard at a right triangle. We still have it, by the way, although it no longer satisfies his famous Theorem since most the hypotenuse was lost during the Great Fire of London. It's too fragile for public view, but @ohabryka will let you see it if you donate more than $1K to Lightcone.

It is so perfect that people who are obviously rationalists by any historical or objective standards would have as a major premise that We Are Not Rationalists, We Are Something New, Formed From The True Cognition. The premise is always denying history.

@deanwball Mazal tov, I guess?

Opus 4.7 just invented a new wife for me

@tyler_m_john As I understand it though, there are different approaches to 2nd order uncertainty (if used at all) - mainly 2nd order probabilities and various types of imprecise probability.

@tyler_m_john 2nd-order uncertainty can be decision relevant. Risk aversion is an example: If ample data shows flood risk 1%, but there's also a war looming, w/ est. chance of home destruction anywhere b/w 0.01%-2% (EV≈1%). For risk averse, war insurance is more valuable than flood insurance.

@tyler_m_john So does that mean you don't actually use higher-order uncertainty in forecasting & decision making except to communicate lack of precision for those who might get hung up about that, plus maybe to help keep track of underlying reasons for updating purposes?

@AryehEnglander 50% chance of 30% chance and 50% chance of 70% chance is equivalent to 50% chance unconditionally. Not that you shouldn't keep track of the underlying reasons so you can update appropriately.

@tyler_m_john Meaning - I'm not sure what you mean by "multiply it out"

@tyler_m_john Not sure what you mean by that exactly. Can you elaborate a bit?

@deanwball Eliezer != Doomers in general (by most common uses of the term these days)

Whenever I say “the doomers express way too much confidence” I inevitably get waves of responses saying “actually rationalists have highly calibrated probabilities.” Does this tweet convey a sense of carefully measured probability? Does it seem uncertain… at all?

have you seen this pattern before? - knows more STEM - knows less about celebrities and sports - worse at following instructions - better coding perf - worse performance at admin/ops - knows more literature - less engaged by pointless brainteasers and needle-in-haystack searches

@sama April 2025: "we fix our model naming by this summer" OpenAI Summer 2025 to now: GPT5 -> GPT5.1 -> GPT5.2 -> GPT5.3 -> GPT5.4 -> GPT5.4-Cyber -> GPT-Rosalind ...So close. They almost had it.

Have AI capabilities accelerated? On 3 out of the 4 AI capability metrics we investigated, we found strong evidence of acceleration, around when reasoning models emerged.

New column: I went to visit @METR_Evals, the 30-person AI nonprofit that makes the Most Important Chart in the World. I learned a lot, but the most striking thing was how soon some of them think AI R&D could be fully automated. (This year!) nytimes.com/2026/04/17/tec…

@MaskedTorah @TheZvi Was that 613 section numbering a deliberate MaskedTorah reference?

@TheZvi Less juicy overall than last time, but I was happy we got to fit in section 6.1.3:

No rest for the wicked, and no rest for anyone else, I suppose.

Our Fairwater datacenter in Wisconsin is going live, ahead of schedule. As the world’s most powerful AI datacenter, it will bring together hundreds of thousands of GB200s into a single seamless cluster. Congrats to all the teams who made this possible!

RT @emollick: AI keeps getting better but the last time the shape of the jagged frontier changed radically was o1 & the Reasoner. A good m…

"For the first time in the history of this war, Ukraine has captured an enemy position using only ground robots and drones. The occupiers surrendered. The operation was carried out without infantry participation and with zero losses on our side." - Zelenskyy.