Text Similarity-based Industrial Dataset Reuse Detection using Sentence Transformer Models under Controlled Perturbations

As more businesses and researchers share data in open marketplaces, keeping track of who is allowed to use that data has become a massive headache. A major problem is “data piracy”—when someone takes an existing dataset, tweaks or rewires the wording just enough to make it look original, and then resells it or uses it without permission.

Standard search tools get tripped up by these surface-level changes. To solve this, our latest study looks under the hood of five modern AI language models to see if they can catch data copycats by looking at the meaning of the text, rather than just the exact words.

We tested a mix of heavyweight and lightweight AI models—including all-MiniLM-L6-v2, Infloat/E5-base, and nli-bert-large—against datasets that were mildly changed, as well as text that was aggressively rewritten.

What we found:

• The AI wins: Every single model we tested was incredibly robust. Even when the text was heavily scrambled, the AI successfully recognized that the core meaning belonged to the original dataset.

• The powerhouses: The larger models (nli-bert-large and Infloat/E5-base) proved to be the most sensitive detectives, catching copies with near-perfect accuracy (matching scores up to 99.7%).

• The speedsters: Smaller, lightweight models (like all-MiniLM-L6-v2) still performed remarkably well, offering a great alternative if you need to scan massive amounts of data quickly without burning through computing power.

Ultimately, this research gives data platforms a clear roadmap for choosing the right AI tool to protect data ownership, balancing the scales between perfect accuracy and computing costs.