Published Paper Details:

Communication is the act of passing ideas from one group to another using mutually recognized signs, symbols and semiotic rules. It is the only medium through which people can exchange their thoughts or convey messages. Deaf and mute people have their own set of signs. Hand gestures are a form of non-verbal communication used by deaf and mute people to communicate with each other and with the outside world. However, the problem with the current world is that most people are not knowledgeable enough to interpret hand gestures or translate them into a language that normal people can understand. Thus, to bridge the communication gap between deaf-mute and normal people, it is important to develop a system that can help them translate gestures into text and/or speech. Creating a robust communication system for the deaf community will help them become more independent and confident. Robust hand gesture recognition has played a significant role in the field of human-computer interaction for a long time, but it is still full of challenges due to many acceptances such as blurred background and hand self-occlusion. With the help of depth information, depth-based methods perform better, but depth cameras are not as widely used and affordable as color cameras. Therefore, we propose two-stage deep convolutional neural network (CNN) architecture for accurate color-based hand gesture recognition. The first phase performs the generation of pseudo-depth hand images from color images, and the second phase recognizes hand gesture classes using the color image and its pseudo-depth hand image. The architecture of the generation phase is based on a picture-to-picture translation network. In the recognition stage, a two-stream CNN architecture with a color image and its pseudo-depth image is proposed to improve the performance of color image-based recognition. Experiments show that our approach significantly improves hand gesture recognition performance in RGB only. The proposed system is user-friendly as it is easy to use and able to create effective and efficient human-computer interaction. A gestural type of communication called sign language that is observed among deaf groups around the world. We designed a deaf-mute verbal exchange device that translates hand gestures into sound massage as an interpreter.