Published Paper Details:

Nowadays Hand Gesture Recognition play an important role as a technique of a development of Enhanced Human Computer Interaction. Upgraded Human Computer Interaction empowers the non-contact framework to cooperate with the PC if there should arise an occurrence of unavoidance circumstance in security and treatment. In this project, Intelligent Virtual Assistant dependent on Hand Gesture and Speech Recognition is proposed to upgrade the client cooperation. The proposed system is divided into four parts: a) hand gesture recognition; b) speech recognition; c) interface object detection; d) command execution. A deep learning method, Convolutional Neural Network(CNN) is used to extract the features of a gesture. To decide these highlights, shading division is utilized to identify the hand; shading pixels can be obtained by removing a specific HSV (tint, immersion, esteem) and applying limit covering to the input image. CNN is also used to extract feature from raw speech signal for efficient speech recognition process. Finally, a Support Vector Machine(SVM) is used to give a more accurate classification of the hand gestures, speech data and GUI Elements. So as to assess the precision of the proposed framework, the accompanying datasets are utilized: Microsoft Kinect, Leap Motion Dataset and Gesture Creative Senz3D for Gesture Detection; Google Speech Commands Dataset for Speech Recognition; pix2code dataset and Sketch2code dataset for GUI component location. The experimental results show that the proposed system can recognize gesture and speech functions with 97% accuracy.