AI Slop Is Already in Your Training Dataset. I Tested Three Ways to Spot It.

In their 2024 International Conference on Machine Learning paper, Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews, Weixin Liang, Zachary Izzo, and their coauthors built a statistical method for estimating how much text in a large collection had been substantially written or rewritten by a large language model, an artificial intelligence system that generates text. They applied the method to peer reviews submitted to four major artificial intelligence conferences after ChatGPT launched. They estimated that 6.5 percent to 16.9 percent of the review text showed signs of substantial AI modification. These were peer reviews written by researchers making careful technical judgments in settings with real submission consequences. That estimate is specific to conference reviews. It shows that substantially AI modified writing appeared in consequential peer review, not how common it is in product reviews, forums, or surveys.

That matters beyond the reviews themselves. A separate 2024 Nature paper, AI Models Collapse When Trained on Recursively Generated Data, by Ilia Shumailov and coauthors, found that repeatedly training a model on output generated by other models can make it lose rare examples from the original data and produce narrower results. The study documents this failure mode under repeated training on generated text.

For years, the standard complaint about data science work has been that most of it is cleaning messy data. A newer problem is that the mess can include fluent text written by a model to sound human. Most data cleaning checks catch missing values, repeated entries, and fields outside an expected range. They do not establish who wrote a paragraph. I wanted to know whether inexpensive checks could flag generated text, and whether removing the reviews they flagged would help a model sort reviews as positive or negative.

I ran two tests using a movie review collection published by Mendeley in 2019. First, I checked which reviews the detectors marked as possibly AI written. A review received that label when its score crossed the chosen cutoff. Then I added 400 reviews generated by two language models to 200 IMDb reviews from the collection. I treated the IMDb reviews as human-written references because the collection identifies IMDb as their source and was published in 2019, before ChatGPT’s public release. I tested three checks on the 600 reviews, measured how many generated and source reviews they flagged, and measured how filtering affected the sentiment model. At a setting that caught 80 percent of generated reviews, embedding density, which measures how closely a review resembles others, also flagged 47 percent of the source reviews. Filtering with the combined score, the average of all three checks, lowered accuracy from 67.5 percent to 57.5 percent on a separate set of 200 reviews. The collection does not verify individual authorship, so this human-written label is an inference from the source and publication date.

What did I actually build?

I first checked the full 1,000 review archive for text that looked AI written. Then I tested three checks on 600 reviews. Perplexity measures how predictable the wording is to a language model. Near duplicate similarity looks for another review with similar wording or meaning. Embedding density measures how close a review is to its five closest matches. The sample included 400 generated reviews with known origins. For each check, I counted how many generated reviews it caught and how many Mendeley reviews, treated as human references, it also marked as possibly AI written. I then measured what filtering did to the sentiment classifier.

For the source reviews, I used 1000 Movie Reviews for Reputation Generation, published by Abdessamad Benlahbib on Mendeley Data on 9 March 2019. The collection description says the reviews were extracted from IMDb, the Internet Movie Database. Since they came from IMDb and the collection was published before ChatGPT launched publicly in November 2022, I treat these reviews as human-written. The dataset does not verify each reviewer’s identity, so this is an inference from the source and date, not an author-by-author check. The collection is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0), which permits reuse and adaptation with attribution and a link to the license. I provide that attribution in the sources. This article uses a subset and does not imply that IMDb endorsed the experiment.

The collection contains 1,000 reviews of 10 films, each with a manually assigned positive or negative sentiment label. I used 200 as human-written reference reviews for training, split evenly between positive and negative, from 2012, A Beautiful Mind, Amadeus, Avatar, and Clash of the Titans. I kept another 200 reviews from five different films separate from training so I could test the model afterward. These test reviews were also split evenly by sentiment and came from Les Miserables, Star Wars Episode I: The Phantom Menace, The Expendables I, The Godfather, and The Matrix Revolutions. Keeping the films separate meant the classifier could not see reviews of a test film during training.

I generated 400 fresh reviews with two language models, gpt-6-astra and gpt-5-nano. One hundred positive and 100 negative reviews came from each model. The prompt asked each model to invent a movie and write a short review with the requested sentiment. It did not show the models source examples or ask them to imitate a particular reviewer. The final training set therefore contained 600 rows: 200 source reviews treated as human references and 400 newly generated synthetic reviews, meaning reviews written by a model.

I use “synthetic” to mean written by a language model. These results apply to the two models, prompts, review collection, and experiment run used here.

How did I try to spot the AI reviews?

I gave each review three scores, then checked how many generated reviews each score caught and how many source reviews it flagged.

First, I measured perplexity with GPT-2. Perplexity measures how predictable a text is to a language model. AI writing can receive a lower perplexity score because it often uses familiar word patterns, but human writing can be predictable too. I treated low perplexity as a warning sign, not proof of AI authorship.

Next, I converted each review into an embedding, a list of numbers that represents some of its meaning and wording. I used the all-MiniLM-L6-v2 sentence embedding model. Near duplicate similarity compares a review with its closest match in that numerical space. A high similarity score means another review has similar wording or meaning.

Embedding density asks a different question. It measures how close a review is to its five nearest neighbors, meaning the five reviews with the closest numerical representations. If AI reviews gather in a tight group, their average distance to those neighbors may be shorter than the distance for varied human reviews.

I also combined the three scores. First, I converted each one to a value between zero and one, with higher values always pointing toward a possible AI flag. Perplexity runs in the opposite direction, so I reversed it before combining the scores. For near duplicate similarity and embedding density, a higher value means the review is closer to other reviews. I then added the three adjusted scores and divided by three. That average is the combined score. It is a warning score, not the probability that a model wrote the review. I used it to test whether averaging several checks worked better than using one on its own.

To compare the methods, I varied the score cutoff and measured recall and the false positive rate. A cutoff is the score above which a review gets flagged. Recall is the share of generated reviews the detector catches. Here, the false positive rate is the share of source reviews treated as human references that it flags. Since the collection does not verify authorship, this rate depends on that assumption. I chose the cutoff that caught at least 80 percent of generated reviews while flagging the fewest of the 200 source reviews in the training set.

What did the audit flag in the original collection?

The combined score marked 444 of the 594 remaining archive reviews, which were not used for training or testing, as possibly AI written. It also marked 158 of the 200 reviews set aside for testing. These are detector flags, not confirmation that a model wrote the reviews.

The combined score averages three checks. They are perplexity, near duplicate similarity, and embedding density.

I chose the cutoff using the 600 review experiment. It was set to catch at least 80 percent of the known AI generated reviews while flagging as few Mendeley reference reviews as possible. I used the same selection rule for the archive audit. Of the 594 remaining archive reviews, 444 scored above the cutoff and the other 150 scored below it.

I downloaded all 1,000 entries in the Mendeley archive. Six were duplicate copies, leaving 994 different reviews. I used 200 reviews for training and kept another 200 separate to test the model. Subtracting those 400 reviews from the 994 distinct reviews left 594 for the audit.

For each check, I chose a score cutoff that caught at least 80 percent of the 400 generated reviews while flagging as few as possible of the 200 source reviews used for training. I then applied the checks to the separate 200 source reviews set aside for testing and the 594 remaining archive reviews, which were not used for training or testing. I scored all 1,394 reviews together because the similarity and density checks compare each review with other reviews. I chose new cutoffs for this larger group. The 0.548 cutoff used later to filter the 600 review training set cannot be reused here. The scores were rescaled across the full group of 1,394 reviews, so the same cutoff would flag a different share of reviews.

The 200 test reviews came from the same Mendeley collection and were treated as human-written references under the source-and-date assumption described above. The combined score marked 158 of them as possibly AI written. This shows that the cutoff is unsuitable for labeling or automatically deleting reviews from the archive.

Which detector flagged the fewest source reviews?

None did well on this collection. Embedding density had the lowest source review flag rate, but it still flagged 47 percent of the reviews treated as human to catch 80 percent of the generated reviews.

Perplexity flagged 98.5 percent of the source reviews at that recall target. Near duplicate similarity flagged 52 percent, and the combined score flagged 73 percent. Averaging the signals did not solve the problem. It made the result worse than using embedding density alone.

At the combined score cutoff used later in the experiment, the detector flagged 320 of the 400 generated reviews and 146 of the 200 source reviews treated as human. It removed 466 rows from the 600 row training set and left 134. So “80 percent recall” did not mean that the filter removed only generated writing. It caught 80 percent of the generated reviews while also flagging nearly three quarters of the source reference reviews.

Which source review scored highest?

The highest scoring source review was a fact heavy entry about Amadeus, not a piece of polished prose.

Here is the full review.

Amadeus is a 1984 American period drama film directed by Milos Forman, written by Peter Shaffer, and adapted from Shaffer’s stage play Amadeus (1979). The story, set in Vienna, Austria, during the latter half of the 18th century, is a fictionalized biography of Wolfgang Amadeus Mozart. Mozart’s music is heard extensively in the soundtrack of the movie.

The film was nominated for 53 awards and received 40, which included eight Academy Awards (including Best Picture), four BAFTA Awards, four Golden Globes, and a Directors Guild of America (DGA) award. As of 2016, it is the most recent film to have more than one nomination in the Academy Award for Best Actor category. In 1998, the American Film Institute ranked Amadeus 53rd on its 100 Years… 100 Movies list.

This review summarizes the film, lists awards, and gives its place on a movie list. Its combined detector score was 0.796, above the 0.548 cutoff. The review reads like a compact reference entry rather than a personal reaction. The score reflects text patterns, not the writer’s identity.

This example shows why a detector score alone is a poor reason to delete a review. Factual, short, formulaic writing can score highly without revealing who wrote it.

Which AI review slipped past the detectors?

The detector also missed this AI generated review about a failing space station.

Here is the full review.

The Last Stop on Neptune somehow turns a murder mystery aboard a collapsing space station into two hours of people arguing beside identical doors. Its detective, Mara Voss, solves every clue by remembering a childhood lesson we’ve never heard before, which is less deduction than screenwriter-assisted cheating. The effects look expensive, but the dialogue sounds like it was borrowed from a motivational calendar. By the time the killer revealed his plan to “murder gravity,” I was rooting for the station to explode. ★★☆☆☆

The combined score was 0.384, below the filtering cutoff of 0.548, so the review was not flagged. The detector caught many generated reviews in this sample but missed this one.

Did filtering make the sentiment classifier better or worse?

In this one run, filtering made the classifier less accurate. Downweighting the flagged reviews also performed worse than leaving the data alone.

The classifier was a logistic regression model. In plain terms, it learned from labeled examples how review wording relates to positive or negative sentiment. It used the same sentence embeddings as the two embedding based checks. I measured accuracy on the 200 source review examples from films excluded from training. Accuracy is the share of those reference reviews the classifier labeled correctly.

I compared four training sets:

• Untouched. All 600 reviews, with no cleaning.

• Filtered. The 134 reviews left after removing every row at or above the combined score cutoff that caught 80 percent of the AI reviews.

• Downweighted. All 600 reviews, but reviews with higher combined scores had less influence during training.

• Source only reference. The 200 Mendeley reviews treated as human examples. This is a comparison point, not a practical filter, because the collection does not verify authorship.

The untouched set reached 67.5 percent accuracy, or 135 correct predictions out of 200. The filtered set reached 57.5 percent, or 115 correct predictions. The downweighted set reached 63.5 percent, or 127 correct predictions. The source only reference reached 68.5 percent, or 137 correct predictions.

The filtered set shrank from 600 reviews to 134 because the cutoff removed source reviews as well as generated ones. The experiment changed both the training data and its size, so it cannot attribute the accuracy drop to removing generated text alone. It does show that a detector score by itself is not a reason to filter. Measure the effect on the task the model needs to perform.

I ran the test once with a fixed random seed, a number that makes the same random selections repeatable. I did not repeat the test across multiple samples or calculate confidence intervals, ranges that show how much an estimate might vary across repeated samples. The accuracy figures therefore describe this run and its 200 test reviews. Repeated runs on new samples would show whether the same pattern holds.

Should you filter, flag, or downweight?

First test your detector on writing from the same source as your production data, including examples with verified authorship when possible. Do not remove reviews just because a detector gives them a high score.

In this sample, the best single signal still flagged nearly half of the source reviews treated as human at the chosen AI catch rate. The combined score performed worse, and using it to filter reduced the training data from 600 reviews to 134. The downstream model then made fewer correct predictions than the model trained on all 600.

If your dataset is large enough, set aside examples with verified authorship and measure false alarms before deploying a filter. Compare the model trained with no filtering against the filtered and downweighted versions. If people can review flagged examples, use that review to decide which items actually need action. If no one will inspect them, a flagging queue is just an automatic filter with an extra step.

What can this experiment tell us?

In this test, the best detector caught 80 percent of the generated reviews while flagging 47 percent of the source reviews treated as human. Removing the flagged reviews cut the sentiment classifier’s accuracy from 67.5 percent to 57.5 percent.

The practical lesson is direct. In this experiment, embedding density caught generated reviews but also flagged many Mendeley reviews used as human references. Filtering with the combined score cut the sentiment model’s accuracy from 67.5 percent to 57.5 percent. Test detector flags against examples with verified authorship, then measure whether filtering improves the task your model must perform. These results apply to this collection, these two models, and this train and test split. The archive audit identifies reviews the checks flagged; it does not establish how many were written by AI.

Reproducing the experiment

The public reproduction repository contains the scripts, the exact 600 training reviews, the separate 200 review test set, and the saved results. Clone it on a computer with Python 3.11 or newer. The first run downloads GPT-2 and the sentence embedding model; both run locally afterward. Clone the repository, enter its folder, install the requirements, and run:

To repeat the audit of the full Mendeley collection, download version 1 from the Mendeley collection page, extract Dataset.rar, and point the audit script to the extracted folder:

The repository also includes the saved output files for comparison. Small numerical differences can occur across operating systems, hardware, and library versions. The Mendeley-derived reviews are an adapted subset of the collection and are attributed in the repository. The audit reports detector flags, not verified authorship.

Sources and dataset attribution

[1] W. Liang, Z. Izzo, et al., Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (2024), International Conference on Machine Learning. Source for the estimate of AI modification in conference peer reviews.

[2] I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, Y. Gal, AI models collapse when trained on recursively generated data (2024), Nature. Background on the risks of repeatedly training on model generated text.

[3] Abdessamad Benlahbib, 1000 Movie Reviews (Review + Attached rating + Sentiment polarity) for Reputation Generation, Mendeley Data, version 1 (2019), DOI 10.17632/38j8b6s2mx.1. The record lists this dataset under CC BY 4.0. I used 400 source reviews for training and testing, sampled by sentiment and separated by film, and audited the other 594 distinct reviews. This is an adaptation of the published dataset. The downloaded archive’s file timestamps are from 2017 to 2019, but the record gives no original posting date or authorship labels for individual reviews.

[4] GPT-2, the open language model used to calculate perplexity.

[5] all-MiniLM-L6-v2, the sentence embedding model used for similarity, density, and sentiment classification.

[6] OpenAI, Introducing ChatGPT, 30 November 2022. Date of ChatGPT’s public launch.