Published Paper Details:

Communication involves conveying thoughts and ideas to others through speech and facial expressions. It is a vital aspect of human life, facilitating interaction with the world. However, for individuals with hearing impairments, communication can be challenging without the aid of devices like battery-operated hearing aids, hand signals, and facial expressions. In this project, an Audio-Visual Speech Recognition (AVSR) system is introduced, leveraging image-processing capabilities in lip-reading to support speech recognition systems. This approach integrates both audio and visual processing, meaning that separate video and audio recognition processes are combined. Our project utilizes a Kannada audio database--a Dravidian language spoken by over 60 million people in Karnataka--to assist those with hearing impairments. The proposed AVSR model comprises three main components: (a) an audio feature extraction mechanism, (b) a visual feature extraction mechanism, and (c) an audio and visual feature integration mechanism. The integration mechanism merges the outputs from both the audio and visual extraction processes to produce final classification results. In this work, the audio feature mechanism is designed using Mel-frequency Cepstral Coefficients (MFCC) and a Random Forest system, while the visual feature mechanism is developed using Haar-Cascade Detection with MATLAB and a Feedforward Neural Network system. These extracted features are then integrated using a hybrid model that includes both Random Forest and Artificial Neural Network (ANN) techniques. The system's performance, in terms of speech recognition accuracy and robustness, was tested using speech data in a clean environment. On average, the final audio recognition rate is 88%, the visual recognition rate is 57%, and the integrated system achieves an overall recognition rate of 86%.