Published Paper Details:

Deaf and mute communities include individuals with partial or complete hearing and speech impairments who primarily depend on sign language for communication. However, interaction with non-signers remains difficult due to limited sign language awareness, creating barriers in education, healthcare, workplaces, and public services. Existing assistive communication systems face challenges such as restricted vocabulary coverage, reliance on predefined gestures, poor real-time performance, limited support for regional sign languages like Indian Sign Language, lack of two-way communication, and inadequate handling of continuous gestures, facial expressions, and environmental noise, which significantly reduces their real-world effectiveness. To overcome these challenges, this project introduces an intelligent, AI-driven communication framework designed to provide seamless two-way interaction between sign-language users and non-signers. The proposed system incorporates a Transformer-Based Gesture Encoder that analyzes continuous hand movements, facial expressions, and body dynamics from live video streams to accurately recognize Indian Sign Language. This enables natural and real-time interpretation of signs without relying on predefined gesture templates, thereby improving recognition accuracy and adaptability in dynamic environments. In addition, the system integrates a Speech Recognition and Synthesis Module utilizing advanced techniques such as RNN-Transducer, Connectionist Temporal Classification, and Deep Neural Networks to convert spoken language into text with high reliability, even under noisy conditions. The interpreted text is further processed by an Avatar Module based on Neural Avatar Synthesis, which generates realistic, human-like sign language animations to visually convey spoken or typed content. Implemented through a web-based interface, the system ensures accessibility, scalability, and ease of use, ultimately promoting inclusive and effective communication across diverse user groups.