Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.
Advertisement
Jackson Ryan is a freelance science journalist in Sydney, Australia.
You can also search for this author in PubMed Google Scholar
Do squirrel surgeons generate more citation impact? The question seems ludicrous, or perhaps the start of a bad joke. But the question, posed by data scientist, Mike Thelwall, was not a joke. It was a test. Thelwall, who works at the University of Sheffield, UK, had been assessing the ability of large language models (LLMs) to evaluate academic papers against the criteria of the research excellence framework (REF), the United Kingdom’s national audit of research quality. After giving a custom version of ChatGPT the REF’s criteria, he fed 51 of his own research works into the model and was surprised by the chatbot’s capability to produce plausible reports. “There’s nothing in the reports themselves to say that it’s not written by a human expert,” he says. “That’s an astonishing achievement.”
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
£14.99 / 30 days
cancel any time
Subscribe to this journal
Receive 51 print issues and online access
£199.00 per year
only £3.90 per issue
Rent or buy this article
Prices vary by article type
from$1.95
to$39.95
Prices may be subject to local taxes which are calculated during checkout
Nature 633, S18-S20 (2024)
doi: https://doi.org/10.1038/d41586-024-02989-z
This article is part of Nature Index 2024 Artificial intelligence, an editorially independent supplement. Advertisers have no influence over the content. For more information about Nature Index, see the homepage.
Thelwall, M. J. Data Inform. Sci. 9, 1–21 (2024).
Article Google Scholar
Liang, W. et al. NEJM AI https://doi.org/10.1056/AIoa2400196 (2024).
Article Google Scholar
Download references
Rise of ChatGPT and other tools raises major questions for research
Rage against machine learning driven by profit
AI’s international research networks mapped
Why AI might be a game-changer for Africa
Artificial intelligence laws in the US states are feeling the weight of corporate lobbying
Partner content: Sum up the science, then prompt new ways of thinking
Partner content: AI can hunt for hidden clues to new drugs in published papers
Partner content: Towards trustworthy, eco-friendly AI
Partner content: Testing toxicity using stem cells and AI
Partner content: AI’s bridges between human knowledge and endless solutions
Partner content: How to harness AI across healthcare
Partner content: AI powers a new era of drug discovery and development
Partner content: Smart AI modelling for precision medicine
Partner content: A robotic foundation model for production
Partner content: Taming AI: A human job
Rising stars in AI use robotics and automation to accelerate their work
Nature Index
AI learns from chromatin data to uncover gene interactions
News & Views
The AI tool that can interpret any spreadsheet instantly
News & Views
Can novelty scores on papers shift the power dynamics in scientific publishing?
Nature Index
Pioneering journal eLife faces major test after loss of impact factor
News
‘Getting paid to review is justice’: journal pays peer reviewers in cryptocurrency
News
Who’s quitting academia? Data reveal gender gaps in surprising fields
News
‘WithdrarXiv’ database of 14,000 retracted preprints launches
News
‘Precocious’ early-career scientists with high citation counts proliferate
News
Xiaoliang Sunney XIE’s Group is recruiting researchers specializing in Alzheimer's disease (AD).
Beijing, China
Changping Laboratory
The School of Engineering (SOE) at Westlake University is seeking to fill multiple tenured or tenure-track faculty positions in all ranks.
Hangzhou, Zhejiang, China
Westlake University
As an Editor, you will support the Chief Editor in managing the Editorial Board and editorial decision-making processes across the journal.
Heidelberg, London or Pune – Hybrid working model
Springer Nature Ltd
Professor openings in mechanical engineering, control science and engineering, and integrating emerging interdisciplinary majors
Nanjing, Jiangsu (CN)
Southeast University, China
At the Faculty of Biological Sciences of Friedrich Schiller University Jena, Germany a W2 Professorship for Integrative Zoology of Invertebrates …
Jena, Thüringen (DE)
Friedrich-Schiller-Universität Jena
Rise of ChatGPT and other tools raises major questions for research
Rage against machine learning driven by profit
AI’s international research networks mapped
Why AI might be a game-changer for Africa
Artificial intelligence laws in the US states are feeling the weight of corporate lobbying
Partner content: Sum up the science, then prompt new ways of thinking
Partner content: AI can hunt for hidden clues to new drugs in published papers
Partner content: Towards trustworthy, eco-friendly AI
Partner content: Testing toxicity using stem cells and AI
Partner content: AI’s bridges between human knowledge and endless solutions
Partner content: How to harness AI across healthcare
Partner content: AI powers a new era of drug discovery and development
Partner content: Smart AI modelling for precision medicine
Partner content: A robotic foundation model for production
Partner content: Taming AI: A human job
An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday.
Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.
© 2025 Springer Nature Limited
About The Author
More Stories
Solving Crime Without Breaking Encryption
Community Snapshot—May
From Refugee to Digital Leader: How Justin Is Helping to Connect Rhino Camp