Published Paper Details:

The ever-expanding use of code-mixed language, mainly on social media platforms, has resulted in challenges for natural language processing (NLP) tasks, due to its irregular writing pattern and the application of multiple languages within a sentence or phrase. Code-mixing of Gujarati-English is increasingly common in multilingual communities, as Gujarati diaspora around the world switch between their mother tongue and English while commenting/twitting/posting their views. In this paper, we present an efficient solution for word level language identification for low resource scenarios by application of fine-tuned version of DistilBERT--a lightweight transformer-based model. Our dataset consists of code-mixed social media comments from YouTube with each word annotated as one of three language tags: Gujarati, English, or Other. It comprises of 77,761 annotated sentences containing 732,917 words, with language labels distributed as 56.06% Gujarati (GJ), 36.77% English (EN), and 7.10% Other (OT). The distinctive part of this work is its fine-tuning process which is entirely conducted using CPU by dividing the training data into chunks of 1000 sentences each. This chunk-based training allows the large dataset to be processed in incremented versions by preserving the optimizer and scheduler states through different iterations. The proposed model gained an accuracy of 97.09%, precision of 97.02%, recall of 97.09%, and F1-score of 97.01%. These results outperform our baseline ML based Random Forest Model which was trained on hand crafted features and achieved accuracy of 91.2%. This proves the effectiveness of transformer-based fine-tuning for language identification in code-mixed contexts.