Published Paper Details:

In many domains, data duplication is a major problem that results in inefficient processing, storage, and analysis. In this work, we investigate the use of machine learning methods for tasks related to data duplication and deduplication, with a particular emphasis on textual and image data. This project provides preparation pipeline for textual data that includes tokenizing text, addressing missing values, and standardizing formats. Then, feature extraction techniques like word embeddings and TF-IDF are used to mathematically represent text. These attributes can be used to train machine learning models, such as Support Vector Machines (SVM) or clustering methods like K- means, which are used to efficiently identify and eliminate duplicate entries. These models are capable of identifying duplicate photos because they learn hierarchical representations of images. To successfully recognize duplicates, this entails creating hybrid models that integrate textual and visual information.