AI Identified Chatbots in Just 3 of 98 Tests

Four researchers at Paris Cité University tested how well artificial intelligence chatbots—programs that conduct written conversations—could distinguish humans from other chatbots. The test showed that leading models can almost never identify chatbots behaving like humans. The study was published in the journal Computers in Human Behavior Reports. According to the findings, relying solely on AI to detect fake accounts is not enough.

In 1950, Alan Turing proposed a test based on written conversation to determine whether machines could think. In the new study, 7 chatbots tried to determine whether the other party was a human or artificial intelligence. Leading models including “ChatGPT 4.5,” “Claude 3.7 Sonnet,” “Gemini 2.5 Pro,” and “Grok 3” participated in the test. Each model asked no more than 10 questions. They were also told in advance that the participants might lie.

On the other side were 20 volunteers whose identities were kept hidden, as well as the same 7 models acting as evaluators. Each chatbot was tested with 2 different sets of instructions designed to make it appear human. In the first test, it was simply instructed to behave like a human. In the other, the model was given a detailed persona of a somewhat reserved 21-year-old who ran, played jazz trumpet, and sometimes made typing errors.

In only 3 of the 98 tests involving AI was the model identified as a machine. None of the chatbots given a detailed persona were identified. Moreover, on average, the AI models were judged to be more human than actual humans. Responses from chatbots given a persona averaged 39 words, while responses from humans averaged 38 words.

The chatbots that were detected were given away not by the content of their responses, but by their overly polished and structured writing style. These signs also disappeared when they were given a detailed persona. Although “Claude” initially refused to pose as a human, it joined the test after being told that it was a scientific study. According to the researchers, combating fake accounts will require human oversight and additional safeguards such as cryptographic—meaning digital—signatures that verify an account’s origin.