Published Paper Details:

This paper introduces a new generative AI method for facilitating human-centric systems: multimodal emotion recognition inferences. Existing emotion recognition models cannot be applied to unseen data well, since the number of emotions is different across datasets. We introduce a synthetic data-based solution to overcome such mismatches in GPT, which uses Generative Adversarial Networks (GANs), diffusion models, and large language models (LLMs) like GPT in conjunction to create expressive samples on visual (facial), auditory and textual modalities. To learn deep neural networks like ResNet-18, BiLSTM and BERT, the synthetic samples are incorporated for training with real-world datasets in the proposed system. This a late fusion strategy using attention but the attention is over the modality cues which tries to select relevant modalities based on context. Experiment results on benchmark datasets (FER2013, RAVDEEDS, and IEMOCAP) show the effectiveness of our framework by achieving state-of-the-art performance with higher emotion classification accuracy, robustness and cross-domain generalization. It yields new information for the design of emotionally intelligent systems that are adaptive, inclusive and capable of engaging in an empathetic dialogue in real-world scenarios.