A Lightweight Hybrid CNN–Multi-Head Attention Framework for Multi-Dataset Facial Emotion Recognition

Abstract

Facial emotion recognition requires a model to identify small local changes in facial appearance while also relating information from spatially separated regions. We developed a lightweight architecture that combines a convolutional neural network (CNN) with multi-head self-attention (MHSA). Three convolutional blocks extract local features from 48 × 48 grayscale images. The resulting 6 × 6 feature map is converted into 36 tokens and processed by a two-head MHSA module before classification. Attention is therefore used to refine intermediate CNN features rather than replace the convolutional backbone. The model was independently trained and evaluated on CK+, FER2013, RAF-DB, and AffectNet. Subject-independent five-fold cross-validation on CK+ produced a mean accuracy of 97.76% and a macro F1-score of 96.94%. Compared with the matched CNN baseline, the proposed configuration improved accuracy by 3.20 percentage points on FER2013, 3.50 points on RAF-DB, and 3.90 points on AffectNet. The ablation results showed that two attention heads performed better than one- and four-head configurations under the fixed embedding dimension. Grad-CAM maps indicated that predictions were influenced mainly by the mouth, eyes, eyebrows, and mid-face. These findings show that refining compact convolutional features with MHSA can improve FER performance with a measured increase from 1.2 to 1.5 million parameters.

Received: 11 April 2026 | Revised: 2 September 2026 | Accepted: 20 September 2026 

Conflicts of Interest

The authors declare that they have no conflicts of interest in this work. 

Data Availability Statement

The data that support the findings of this study are openly available in the CK+ dataset at https://www.jeffcohn.net/resources/, reference number [28]; in Kaggle at https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data; in the RAF-DB dataset at http://www.whdeng.cn/RAF/model1.html; and in the AffectNet dataset at https://mohammadmahoor.com/pages/databases/affectnet/.

Author Contribution Statement

Ayman Odeh: Conceptualization, Methodology, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Supervision, Project administration. Munther Odeh: Methodology, Software, Investigation, Data curation, Writing – original draft. Nada Odeh: Methodology, Software, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization.