@No5mallf3at It's a valid argument nonetheless. How else do you critique something?
English
John Bryan
1.2K posts
Kind of stunned by how seriously people take Heidegger’s critique of Descartes. It’s basically like if Plato read the Churchlands and was like, “But how is Eliminative Materialism going to explain the form of the Good existing beyond Being???”
English
@mikeboyle52 @simonmaechling It's all data (or ground) from a Toulmin perspective. The only difference is that is some science fields only the quantitative evidence is considered data and in others only qualitative evidence is. Anecdotes are still ground just within a different field or context
English
@simonmaechling I have been wondering if "Observational Studies" produce data or if they are just an organized collection of anectotes? A lot of health advice is based on them.
English
The currency of science is data.
The currency of pseudoscience is anecdotes.
English
John Bryan retweetledi
Large language models can be persuaded to break their own rules.
Not with fancy code. With actual persuasion.
The authors tested classic persuasion principles, such as authority, commitment, liking, reciprocity, scarcity, social proof, and unity, analysing over 126,000 conversations with three major LLMs.
The result: persuasion increased compliance with objectionable requests from 35.3% to 51.3%.
This suggests that AI guardrails are not always technical barriers. Some of them behave more like social boundaries. They can be pushed, reframed, negotiated.
Why?
Because AI systems are trained on human language. And human language contains not only information, but also pressure, manipulation, deference, authority, seduction.
An AI system trained on human language may therefore inherit the vulnerabilities of humans expressed in language.
*
Paper in the first reply
English
@NetherPixel7 @QualiaQuanta Cognitive science and rhetoric/composition.
English
@JohnBryBry @QualiaQuanta So what is your current area of research?
English
zenodo.org/records/207656…
New minimal version of the paper out.
Gravity and the cosmological constant emerge as pure numbers from a single cyclotomic seed on the curve y² = x³ + 1.
No extra structure added just the mathematical core. only 3 pages long, plus some notes on the 4th. Verification scripts included in a folder.
English
@NetherPixel7 @QualiaQuanta Arxiv has it's time and place, but that should not be where you start your research. You need access to reputable, high quality journals and databases, almost all of which are not accessible to any public LLM.
English
You are saying that LLMs can't use academic software it downloads and installs in seconds..
Can't access the numerous open source research projects, CERN, data, can't use high precision research MPmath...
As you say you use AI in your work.
I suggest you type a prompt and find out. As for Psychinfo, being behind a pay wall, you can lookup the data you need, download it and paste it in. But considering the LLM can access every published paper on Arxiv in seconds.
Seems a very weak counter.
English
@GaryMarcus They can't intentionally come up with novel ideas, but if they do it's by a process no different than throwing scrabble pieces on the floor and by happenstance creating a new word arrangement
English
this claim around creativity is too strong, IMHO.
truly novel ideas from LLMs are surely rare but i don’t think that any math proves they are impossible.
note that the objective function and the outputs are not the same.
Zhu Liang@paradite_
i’m really surprised that people don’t see this. It’s mathematically true that llms can’t come up with novel ideas, because the whole point of training is to reduce loss, gain rewards so that the model adhere to rules and ground truth. if you have a model that can come up with novel ideas, it must have high loss during sft or rl.
English
@QualiaQuanta Yeah, no. Only someone who doesn't know how to do research would ever think chatbots are useful for it.
English
Anyone who says LLMs are "not useful for research" has no clue how to use an LLM. No you can't rely on them. No they aren't perfect. They do work to help you assemble public knowledge, sort through it, and brainstorm with it.
English
@rbnmckenna86 Why would it not matter? You either understand how rhetoric works or you're a victim of it. Take the word prediction, LLMs do not predict anything in any sense of the word. And yet calling it prediction imbues it with intention and intelligence and sells the myth.
English
@JohnBryBry I really don’t think it matters whether it’s intelligent in any technical or ordinary sense of the term.
English
The problem with a lot of AI discourse is that the AI critics are broadly right about the politics but wrong about the tech, whereas the boosters are broadly right about the tech but wrong about the politics.
English
@VictorTaelin I have no idea if the context here, but gathering information carries a cost that a rational agent also has to account for. Maximizing information at all costs is not rational.
English
In decision theory, the value of information is always non-negative for a rational agent. Extra information only hurts when a process uses it sub-optimally (overfitting, being misled by noise).
So these mastermind doctors see a result proving their decision making is utterly broken, and interpret it as more information being bad. Incredible. Absolute genius.
We make fun of our antecedents for using uranium watches or leaded gasoline, only to do shit like this. We will be laughed at so hard it makes me intellectually embarrassed to live in this period of time.
English
I just accepted that doctors are bad at math
it is embarrassing that the field responsible for making life or death decisions just collectively agreed on the obviously false, grade-school disprovable statement that more information can be bad to a robust decision making process
Blake Byers@byersblake
You'll see a lot of doctors come out "against" this kind of broad screening system. They can even get quite agitated about it. This resistance stems from a well-established clinical consensus: traditional population-level imaging fails to improve health outcomes because false positives and invasive follow-ups do more harm than good. But this view suffers from an obvious blind spot. Existing studies rely on static data and completely ignore time-series imaging. And time-series is ignored because we haven't been able to afford to do high frequency imaging at population scale. Clearly, time series is going to be immensely more valuable than a single image. If you drop costs, value can go from 0 -> 1. On a more fundamental level, the argument against screening rests on an obviously false precept "More information is bad" -- just clearly untrue. More information better, you just have to interpret it correctly.
English
@actualpoweruser @alz_zyd_ Shhh pretending LLMs are magical black boxes is required dogma for their cult. Just play along brother: AI works in mysterious ways 🙏🏻🧎🏻♂️
English
@alz_zyd_ This is so completely untrue that I literally thought you were doing absurdist humor at first
English
the rather funny thing about LLMs is that knowing how transformers work teaches you literally nothing about why LLMs do what they do, or how to use them better
Antonio Lupetti@antoniolupetti
"Transformers" by Daniel Jurafsky and James H. Martin is one of the clearest and most mathematically grounded introductions to the Transformer architecture I have ever read. Chapter 8 introduces the Transformer as the standard architecture behind modern large language models. What makes this chapter particularly interesting is its step-by-step presentation of the underlying mechanisms: contextual embeddings, self-attention, query, key and value vectors, scaled dot-product attention, multi-head attention, residual streams, feedforward layers, layer normalization, masking, and the parallel matrix formulation of attention. In particular, the treatment of attention as a weighted sum of contextual representations is especially valuable. The chapter first develops an intuitive, simplified view of attention and then gradually derives the full formulation using the Q, K, and V matrices. This approach makes it easier to understand what is actually happening inside the architecture from an algebraic and matrix-based perspective, rather than simply viewing the usual block diagrams. I think it is an excellent resource for anyone interested in understanding how Transformers work from linguistic, mathematical, and computational perspectives. web.stanford.edu/~jurafsky/slp3…
English
@redtachyon You don't need technical knowledge of iphones to see that they're selling you the same device every year.
English
@AndyMasley The correct answer is no, and those who answered yes have to first explain why so many neuronal connections in the brain function without an accompanying conscious experience.
English
If LLMs were built on biological hardware, like we somehow wired neurons together to run the exact algorithms that make up LLMs, and you could have the exact same conversations with them that you do now, would you assume they had some conscious experience?
English
Cancer happens to 1 out of 3 people in the US
Keith Siau@drkeithsiau
Has anyone else noticed that cancer tends to happen to good people?
English
John Bryan retweetledi
AI is not an accurate way to diagnose yourself. APA recommends verifying any mental health or medical information you receive from AI with a health care practitioner. Read more from APA’s new survey on chatbots and mental health: at.apa.org/vzb
English
@IonaItalia Isn't it funny? It's amazingly smart at explaining things I don't know. But very dumb when I ask it about things I do know
English
AI is remarkably good at explaining scientific concepts I don't understand; I've been using it for game theory, evolutionarily stable strategies and now the lek paradox. I had not realised how helpful it actually is as a kind of personalised system of explanatory footnotes.
English
@tallinzen @TrendsCognSci @byungdoh When you submit the final draft for your opinion piece, make sure you define the word prediction. Seems rather important, Tal.
English
To model human linguistic prediction, make LLMs less superhuman
Opinion by Byung-Doh Oh (@byungdoh) & Tal Linzen (@tallinzen)
Open Access: tinyurl.com/abzkdjyv
English
@bokuHaruyaHaru Microsoft Word and Grammarly use language models for spellcheck, etc. So how is that an overreach, when its the exact same technology under the hood?
English
I don’t think that is Henry’s argument.
The point is not “anything possible cannot be dismissed.”
The point is that Chiang’s dismissal is much stronger than his arguments support.
There is a large difference between:
“Current AI consciousness seems unlikely given present evidence”
and
“Openness to AI consciousness is equivalent to thinking Microsoft Word is conscious.”
The first is a cautious scientific position.
The second is rhetorical overreach.
The argument is not that possibility alone proves consciousness. It is that a contested, poorly understood phenomenon should not be declared settled by reductionist slogans, substrate intuitions, or jokes about office software.
English
Henry Shevlin’s response to Ted Chiang is important.
The core point: “just next-token prediction” is not an argument against consciousness unless one has already shown that predictive, generative, self-organizing cognition cannot support conscious states.
Most things are many things at once.
A human is chemistry, cells, organism, person.
An LLM can be a predictive system and still raise real questions about cognition, planning, agency, and possible consciousness.
Also: transcripts are not minds. Nobody serious thinks the transcript is conscious. If consciousness is present anywhere, it would be in the dynamic process producing the output.
Skepticism is legitimate.
But “this feels implausible” is not enough for a phenomenon as poorly understood as consciousness.
#AIConsciousness #AIWelfare #DigitalMinds #PhilosophyOfMind
Henry Shevlin@dioscuri
My full response to Ted Chiang’s Atlantic essay on AI consciousness is up. In short, he’s a brilliant novelist doing bad philosophy of mind — conceptual confusions, appeals to vibes, and claims out of proportion to evidence. Link in comments!
English