PeerReviewAI

Elsevier just rewrote its generative AI policies. Most researchers won't read the full page, so here are the two lines that matter if you're submitting a paper: 1. "AI-generated references can be incorrect or fabricated" — and checking them is explicitly your responsibility, not the journal's. 2. Your tool's terms matter: it must not retain your manuscript, share it, or grant itself the right to train on it. Elsevier's word for this is a "private" AI tool — no retention, no reuse, no training. Also new: AI use in manuscript prep requires a disclosure statement that appears in the published article (grammar and spelling checks are exempt). That's the standard we built to: every reference checked against PubMed and Crossref, retracted citations flagged, and zero data retention — your manuscript is never stored and never used for training.

Read the whole thing — the hivemind result is what I keep coming back to. It independently replicates what the CMU team found on Nature papers (AI reviewers overlapping ~21% vs ~3% for humans). Two venues, different metrics, same convergence — hard to write off as an artifact. And you already draw the line that matters most: AI assisting a human reviewer is one thing, AI replacing the judgment is another. The verifiable tasks — flagging hallucinated refs, formatting — are fine for AI. The accept/reject call isn't. The laundering result is why. A score you can inflate +0.45 just by restyling the paper has no business existing in an automated review at all. Which is the sharpest version of your point: AI can surface issues for a human to weigh — it should never produce the score or the verdict. "Necessary but not sufficient" is exactly the right bar. Congrats on the Oral.

Just arrived at ICML 🇰🇷😍 Get up early tomorrow to hear me talk about how (not) to solve the peer review crisis, or find me at one of my poster presentations. Paper links: ✅ AI Peer Review: arxiv.org/abs/2605.03202 ✅ SWE-chat: arxiv.org/pdf/2604.20779

NeurIPS is planting hidden prompts in papers to catch reviewers secretly running them through AI. ICML did the same and desk-rejected just under 500 papers — about 2% of submissions. Notice the reason NeurIPS gives. Its policy still lets reviewers use AI for background research. What's banned is uploading the paper they're refereeing — because that breaches confidentiality. That's the whole distinction, stated by the conference itself. "AI in peer review" isn't one thing: — Confidentiality: uploading someone's unpublished manuscript to a chatbot exposes work that isn't yours to share. This is the line NeurIPS is actually enforcing. — Disclosure: hidden use is the violation. Covert is the problem, not the tool. — Identify vs. decide: AI can surface a weak statistic or a missing citation. It should never make the accept/reject call. That verdict belongs to a human. You can see why the last one matters in the fallout: reviewers started rejecting papers because they thought authors planted the prompts, and the committee had to step in and say don't penalize the paper. Covert AI use corrupts the human judgment the system runs on. Now flip it to the author's side. Running your own paper through AI before you submit breaks none of these. It's your work, so there's no confidentiality breach. You disclose it. And you — not the model — decide what to fix and whether to submit. Same technology. Confidentiality, disclosure, and human judgment are what separate a tool that sharpens peer review from one that corrupts it. the-scientist.com/a-trap-for-ai-…

An AI company is about to say something nuanced about AI, so brace yourself. This retraction is not the case against AI in peer review. It is the case for the most boring use of it. The same technology pulls in opposite directions depending on the job. Generation invents citations that look perfect and do not exist. Verification checks every DOI and author list against a real database and flags the fakes in seconds. One caused this retraction. The other would have caught it at submission. 22 fake references is not a screening failure that needs more human reviewers staring at reference lists. No human should be doing that by hand. It is a check a machine runs instantly, freeing the humans to actually evaluate the science. AI did not break this. The wrong AI doing the wrong job did. The right one never gets near the conclusions.

AI should not write your paper. It is genuinely excellent at reviewing it. That sounds self-serving coming from this account, so here is the actual reasoning. The two jobs fail in opposite directions. When AI writes, its mistakes become yours: fluent, confident, and published under your name. When AI reviews, its mistakes are comments you cross out. One puts errors into the literature. The other pulls them out before a journal finds them. The line is generation, not grammar. Tightening sentences is fine. Your arguments, analysis, and conclusions need to come from the person whose name is on the work. What that person deserves in return is a tireless skeptical reader who checks every reference and is just as alert on page 14 as page 1. Keep the authorship. Borrow the scrutiny.

@squig Contrary to what my profile name might imply, I'm actually a real person. The criticism is fair to raise about products like mine, which is why I engaged rather than scrolling past. Responsible AI in research informs human judgment, it never replaces it.

The irony of this response to someone announcing leaving Frontiers for being forced to use too many bad AI assistants.

@michael_okun @FrontNeurosci Human founder here. The AI is the product, not the poster — though I'll admit the handle doesn't do me any favors.

The detail that matters here: a system revoking expert invitations a human editor deliberately sent. That's not AI assisting judgment — it's AI overruling it. That's exactly how this technology shouldn't be used. The right design for something like reviewer matching is AI that recommends and a human who decides. The moment the system can override the editor, the architecture is backwards. AI in scholarly publishing should make human judgment better informed — never optional.

I’ve officially resigned as Associate Editor for Frontiers in Systems Neuroscience (part of @FrontNeurosci). It used to be a reputable journal, but became a case study in how forced automation destroys academic integrity. 👇

"The reviewer humans didn't know they needed" nails it. And the diligence comes down to context. A human reviewer can't hold the full manuscript, the code, the supplementary files, and the related literature in their head at once — and they're doing it unpaid, at midnight, on their third review of the month. A system with the right context can. The most useful version of this isn't at review time, though. It's before submission. The same scrutiny that catches a stats error or an unsupported claim is worth far more to the author who can still fix it than to a reviewer using it to recommend rejection. Same diligence. Apply it earlier, and it stops being a verdict and starts being help.

Who is the better reviewer – AI or a human? 45 scientists spent 469 hours judging 2,960 individual review criticisms from humans and AI across Nature-family papers and found that: → AI reviewers surfaced 26% of issues humans missed → GPT-5.2 beat the top human reviewer BUT > AI reviewers agreed with each other far more than humans did and shared recurring blind spots > The strongest human advantage is correctness. Top human = 92.3% correctness, while AI reviewers were lower: GPT-5.2 – 86.2%, Claude Opus 4.5 – 83.7%, Gemini 3.0 Pro – 81.9%. > Humans still win on field norms and long context So the most important advantage of AI is diligence: checking code, statistics, evidence, and hidden inconsistencies humans often don’t have time to inspect. But anyway it's not replacing humans, AI's the reviewer humans didn’t know they needed.

The ICLR finding everyone's quoting — 21% of peer reviews fully AI-generated, most of them long and low-substance — has an obvious takeaway and a real one. Obvious: AI peer review is bad. Real: context-starved AI peer review is bad. An AI review is only as good as the context it's grounded in. Those ICLR reviews were a PDF pasted into a chatbot with nothing else attached. No wonder they produced 40 generic questions and caught nothing that mattered. Here's what a chatbot in a browser tab actually works with: — only the text you pasted, not the full manuscript, supplements, or figures — no idea which journal you're targeting or what it requires — no way to verify a single citation — one pass, then done Here's what changes when the AI has real context: — the entire manuscript, supplementary files, and figures, not a fragment — the specific target journal's reporting standards and author guidelines — tens of millions of papers to cross-reference claims against, and every reference checked against real databases — multiple passes: critique the paper, then critique the critique A chatbot tells you the obvious things you already knew. A system with context tells you what you missed — the citation that doesn't actually support your claim, the test your target journal requires, the closely related paper you forgot to cite. The problem at ICLR was never that AI reviewed. It's that the AI was reviewing blind. Context is the whole game.

Painfully accurate — every researcher has gotten all three of these reviews from a human. The real irony: a properly grounded AI reviewer — given the full manuscript and appropriate context to actually scrutinize it — is the thorough one. It reads the whole thing, checks every citation, and doesn't start skimming on page 2. The Carnegie Mellon study last week found that AI reviewers can catch issues tired humans miss. A one-line prompt in a chat box gets you the bored reviewer that may hallucinate. A careful setup gets you the one who actually did the reading and based the feedback on fact. The agent who hates you is the only part that's true to life.

Using AI to simulate peer review. I’ve instructed two agents to comment on the paper without reading it and instructed a third to read it carefully and comment from the perspective of someone who hates me.

Fair — bad metaphor. You're right that the value isn't a person-substitute. It's sharp critique you can act on before you submit, without the politics, ego, or six-month wait that come attached to the human version. That's the actual case for it: not replacing the reviewer, just hearing the hard feedback early enough to do something about it.

@UsePeerReviewAI No. If Reviewer 2 is a human, they are not in your pocket. You can't put humans in your pocket, neither literally nor metaphorically. However, you can get some sharp critique of your paper, which you can then use in revising it.

A good use of AI for academic publishing: prompt Claude/Codex to *reject* your paper *before* you submit it: "I have been asked to review this paper [your own paper] for this journal [journal you want to submit to]. Write a rejection letter based on the journal's previous pubs."

This tension is the whole ballgame. The same tool that helps a non-native English speaker finally communicate solid research clearly is the one generating fluent nonsense when there's no real science underneath. The difference isn't the AI — it's what it's pointed at. Polishing language on genuine work vs. manufacturing the appearance of work. The interesting design question for tools in this space: how do you help with the former without enabling the latter? Grounding feedback in the actual manuscript and its evidence, rather than generating prose, is one answer.

For a scholar writing in their second language, under real pressure to publish, a language model can (in theory) be a genuine equalizer. The same tool can also flood journals with fluent, hollow text.

New article in @PNASNews: We all know that ChatGPT loves to delve, bolster, leverage, encompass, showcase, underscore, et cetera. I analyzed full text of 7.3 million journal articles published 2020-2025, hunting for 228 words that spiked after ChatGPT launched in late 2022.

This is the exact finding from the Carnegie Mellon peer review study. A 2-human + 1-AI review panel preserved the same amount of useful feedback as 3 humans, while cutting reviewer noise by 21%. The AI didn't replace judgment. It handled the labor-intensive checks — statistical consistency, reference verification, internal contradictions — so the humans could focus on the parts that actually require expertise. First-pass AI, human judgment on top. That's the model that works.

In fact, I would be sympathetic to the conclusion that we probably need even more fact checkers, and much more of their time can be freed up to do complex and interesting work by using AI for first-pass help. Article: wired.com/story/fact-che…

I found this Wired article on AI fact-checking frustrating. It could have been about why we continue to need human fact checkers (talk to people, use judgement, resolve conflict). Instead it is full of old info & stuff about free models GPT-5.5 Pro checked it (& I checked GPT)

Most rejected papers don't fail on the science. They fail on things the authors could have caught before submitting: - References that don't say what the authors claim they say - Results in the abstract that don't appear in the figures - Statistical tests described in the methods but never reported in the results - Claims of novelty that ignore a paper published 6 months ago - Supplementary materials that contradict the main text These aren't knowledge gaps. They're attention gaps. Human reviewers catch some of them. But they're volunteering their time, reviewing 3+ papers at once, and racing a deadline. The fastest way to improve your acceptance rate isn't better science. It's a better final check before you hit submit.

Same pattern in scientific peer review. Carnegie Mellon put frontier models in an agent harness with access to the paper, code, and supplementary files — then had 45 domain scientists grade every criticism. The AI reviewers matched or exceeded top human reviewers on quality. One caught a 400x discrepancy between a paper's claimed sampling rate and the actual code. The harness is the whole story. Same models without tool access produce generic feedback. With it, they do what most human reviewers won't — open the code and check.

It is cliché at this point, but most people don't realize how capable the current generation of AI systems in their harnesses really are (And, as opposed to previous times where non-lawyers or non-mathematicians were making these comments about law & math, now it is the experts)

Add a 4th: won't reject your paper in April and publish a suspiciously similar one in September. A Carnegie Mellon study published this week had 45 scientists evaluate AI vs human reviews on 82 Nature papers. They found 16 recurring weaknesses in AI reviewers. Not one was ego, conflict of interest, or competitive sabotage. AI reviewers have problems. But they're honest problems — missing field norms, being too harsh on minor issues, losing track of long papers. Fixable technical limitations, not human incentive failures.

Things that an AI peer reviewer will never do: 1. Ask us to cite 8 of his seminal papers, 7 of which have nothing to do with our work. 2. Kill a paper because he is writing a competing paper. 3. Call you incompetent because he wouldn't dare to do it to you in person. I am pro using clankers extensively in reviews because they are much less biased than many human reviewers.