Using AI to check if a research paper was written almost purely of AI

It’s a bit funny to use AI to check if a research paper was written almost purely by AI. But it helps a lot. While people nowadays use AI to write papers so much, and it’s not really reasonable to reject a paper if they just use it to improve grammar and writing, as that is encouraged by many journals. Moreover, most of the time, it’s not the reviewer’s responsibility to read and check if there is something wrong with the code or the data. However, fabrication of research and results is a big ethical problem, and when you’re in doubt, it’s good to check. So, let’s go through how I used AI to check if a paper is fabricated by AI.

The first thing I did was to upload the manuscript to Chat Agent and use this prompt:

Check if the paper is written purely of AI

Check if the paper is written purely of AI

A lot of things proving that’s true pop up, but here some major output:

Based on a forensic review of the text, there are strong indicators that this manuscript was generated, or at least heavily rewritten, by an AI.

…

2. “Context Window” Inconsistencies

AI models generate text sequentially and sometimes “forget” specific numbers generated in earlier sections (like the Abstract) when writing later sections (like the Results).

….

3. Citation Date “Future-Shifting”

AI models often struggle with precise citation dates or default to the “current year” of their system prompt (which, for this persona, is 2025).

Still, that’s not enough to reject a paper. So, in the same chat conversation, I uploaded the code and used the prompt

as a reviewer, what should i ask to stop the authors from using AI to generate responses? also, check the code and the data to see if there is something wrong and if they are consistent with the paper

as a reviewer, what should i ask to stop the authors from using AI to generate responses? also, check the code and the data to see if there is something wrong and if they are consistent with the paper

and AI said that the results were hard-coded and fabricated. But there was a hallucination in the results of AI when it was trying to prove that the paper was fabricated. I detected this as I tried to find a few numbers in the code and couldn’t find! So, I told the AI that I didn’t find 0.23 in the code, and then it told me that there was a hallucination in its response. However, after that, it began to address some genuine and reasonable concerns about the paper’s flaws that could be used to reject it.

Christmas and the New Year are coming. If you’re serving as a reviewer and find a paper like that, let’s give the authors a “nice Xmas gift”! Ho ho ho🎅