Published Paper Details:

In a world where communication barriers often isolate individuals with hearing impairments, bridging the gap between spoken language and sign language becomes imperative. This project proposes a novel solution aimed at facilitating seamless communication between the deaf and hearing communities through a multimodal interface. The project focuses on developing two key functionalities: converting audio inputs into sign language and translating sign language gestures into audio and text outputs. To achieve this, the system integrates advanced machine learning algorithms, computer vision techniques, and natural language processing models. For the audio to sign language conversion, the system utilizes speech recognition to transcribe spoken language into text, which is then translated into sign language gestures using a deep learning-based sign language generation model. The generated sign language animations are displayed in real-time through an avatar or a robotic interface. Conversely, for sign language to audio and text translation, the system employs computer vision algorithms to detect and recognize sign language gestures from video inputs. These gestures are then interpreted into text using machine learning classifiers trained on sign language datasets. Finally, the text is converted into spoken language using text-to-speech synthesis for audio output and displayed alongside the animations for text output. The proposed system offers a comprehensive solution for bidirectional communication between the deaf and hearing communities, facilitating real-time interaction without language barriers. It has the potential to enhance accessibility in various settings, including education, healthcare, and public services, ultimately fostering inclusivity and equality for individuals with hearing impairments.