Early Fusion of CNN Features for Multimodal Biometric Authentication from ECG and Fingerprint Using MLP, LSTM, GCN, and GAT
DOI:
https://doi.org/10.52436/1.jutif.2025.6.6.5299Keywords:
Deep Learning, ECG, Early Fusion, Feature Extraction, Fingerprint, Multimodal BiometricAbstract
Traditional authentication methods such as PINs and passwords remain vulnerable to theft and hacking, demanding more secure alternatives. Biometric approaches address these weaknesses, yet unimodal systems like fingerprints or facial recognition are still prone to spoofing and environmental disturbances. This study aims to enhance biometric reliability through a multimodal framework integrating electrocardiogram (ECG) signals and fingerprint images. Fingerprint features were extracted using three deep convolutional networks—VGG16, ResNet50, and DenseNet121—while ECG signals were segmented around the first R-peak to produce feature vectors of varying dimensions. Both modalities were fused at the feature level using early fusion and classified with four deep learning algorithms: Multilayer Perceptron (MLP), Long Short-Term Memory (LSTM), Graph Convolutional Network (GCN), and Graph Attention Network (GAT). Experimental results demonstrated that the combination of VGG16 + LSTM and ResNet50 + LSTM achieved the highest identification accuracy of 98.75 %, while DenseNet121 + MLP yielded comparable performance. MLP and LSTM consistently outperformed GCN and GAT, confirming the suitability of sequential and feed-forward models for fused feature embeddings. By employing R-peak-based ECG segmentation and CNN-driven fingerprint features, the proposed system significantly improves classification stability and robustness. This multimodal biometric design strengthens protection against spoofing and impersonation, providing a scalable and secure authentication solution for high-security applications such as digital payments, healthcare, and IoT devices.
Downloads
References
V. M. Praseetha, S. Bayezeed, and S. Vadivel, “Secure fingerprint authentication using deep learning and minutiae verification,” Journal of Intelligent Systems, vol. 29, no. 1, pp. 1379–1387, Jan. 2020, doi: 10.1515/jisys-2018-0289.
J. S. Mane and S. Bhosale, “Advancements in biometric authentication systems: A comprehensive survey on internal traits, multimodal systems, and vein pattern biometrics,” Revue d’Intelligence Artificielle, vol. 37, no. 3, pp. 709–718, Jun. 2023, doi: 10.18280/ria.370319.
G. Kang, J. Park, and Y. G. Kim, “Continuous Behavioral Biometric Authentication for Secure Metaverse Workspaces in Digital Environments,” Systems, vol. 13, no. 7, Jul. 2025, doi: 10.3390/systems13070588.
S. S. U. Hasan, A. Ghani, A. Daud, H. Akbar, and M. F. Khan, “A Review on Secure Authentication Mechanisms for Mobile Security,” Feb. 01, 2025, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/s25030700.
I. Alkhwaja et al., “Password Cracking with Brute Force Algorithm and Dictionary Attack Using Parallel Programming,” Applied Sciences (Switzerland), vol. 13, no. 10, May 2023, doi: 10.3390/app13105979.
P. Kumar, Y. Dogra, N. Bansal, and C. Jhanjeri, “Biometric system-A Review on its Security.” [Online]. Available: www.ijfmr.com
Z. Unnisa et al., “Threats and Mitigation Strategies for Electroencephalography-Based Person Authentication,” 2025, John Wiley and Sons Ltd. doi: 10.1155/ijta/3946740.
M. Al Alfi, P. Peris-Lopez, and C. Camara, “Enhancing biometric identification using 12-lead ECG signals and graph convolutional networks,” Front Digit Health, vol. 7, 2025, doi: 10.3389/fdgth.2025.1547208.
O. D’Angelis, L. Bacco, L. Vollero, and M. Merone, “Advancing ECG Biometrics Through Vision Transformers: A Confidence-Driven Approach,” IEEE Access, vol. 11, pp. 140710–140721, 2023, doi: 10.1109/ACCESS.2023.3338191.
V. Agrawal, M. Hazratifard, H. Elmiligi, and F. Gebali, “Electrocardiogram (ECG)-Based User Authentication Using Deep Learning Algorithms,” Diagnostics, vol. 13, no. 3, Feb. 2023, doi: 10.3390/diagnostics13030439.
N. Ammour, Y. Bazi, and N. Alajlan, “Multimodal Approach for Enhancing Biometric Authentication,” J Imaging, vol. 9, no. 9, Sep. 2023, doi: 10.3390/jimaging9090168.
A. K. Gona and S. M., “IoT-based multimodal liveness detection using the fusion of ECG and fingerprint,” International Journal of Pervasive Computing and Communications, vol. 20, no. 4, pp. 465–477, Nov. 2024, doi: 10.1108/IJPCC-10-2021-0248.
S. U, K. P. K, S. Prabhu, and V. C. Nayak, “Multimodal biometric authentication: a novel deep learning framework integrating ECG, fingerprint, and finger knuckle print for high-security applications,” Engineering Research Express, vol. 7, no. 1, p. 015207, Mar. 2025, doi: 10.1088/2631-8695/ad9aa0.
R. Garg, P. Pathak, and M. P. Singh, “A multimodal biometric recognition system based on Fingerprints, Iris and ECG via Swin Transformer and CNN Model,” Systems and Soft Computing, vol. 7, p. 200369, Dec. 2025, doi: 10.1016/j.sasc.2025.200369.
R. Srivastava, R. Tomar, A. Sharma, G. Dhiman, N. Chilamkurti, and B.-G. Kim, “Real-Time Multimodal Biometric Authentication of Human Using Face Feature Analysis,” Computers, Materials & Continua, vol. 69, no. 1, pp. 1–19, 2021, doi: 10.32604/cmc.2021.015466.
S. A. El-Rahman and A. S. Alluhaidan, “Enhanced multimodal biometric recognition systems based on deep learning and traditional methods in smart environments,” PLoS One, vol. 19, no. 2 February, Feb. 2024, doi: 10.1371/journal.pone.0291084.
D. S. Ametefe et al., “Enhancing Fingerprint Authentication: A Systematic Review of Liveness Detection Methods Against Presentation Attacks,” Journal of The Institution of Engineers (India): Series B, vol. 105, no. 5, pp. 1451–1467, Oct. 2024, doi: 10.1007/s40031-024-01066-3.
H. Ghalib Muhammad and Z. Ali Khalaf, “A survey of Fingerprint Identification System Using Deep Learning,” International Journal of Computing and Digital Systems, vol. 17, no. 1, pp. 1–17, Jan. 2025, doi: 10.12785/ijcds/1571022983.
T. M. C. Pereira, R. C. Conceição, V. Sencadas, and R. Sebastião, “Biometric Recognition: A Systematic Review on Electrocardiogram Data Acquisition Methods,” Feb. 01, 2023, MDPI. doi: 10.3390/s23031507.
H. G. Muhammad and Z. A. Khalaf, “Fingerprint Identification System based on VGG, CNN, and ResNet Techniques,” Basrah Researches Sciences, vol. 50, no. 1, p. 14, Jun. 2024, doi: 10.56714/bjrs.50.1.14.
D. A. Alduwaile and M. S. Islam, “Using convolutional neural network and a single heartbeat for ecg biometric recognition,” Entropy, vol. 23, no. 6, Jun. 2021, doi: 10.3390/e23060733.
R. M. Jomaa, H. Mathkour, Y. Bazi, and M. S. Islam, “End-to-end deep learning fusion of fingerprint and electrocardiogram signals for presentation attack detection,” Sensors (Switzerland), vol. 20, no. 7, Apr. 2020, doi: 10.3390/s20072085.
S. Artabaz and L. Sliman, “Feature fusion and selection using handcrafted vs. deep learning methods for multimodal hand biometric recognition,” Sci Rep, vol. 15, no. 1, p. 29237, Aug. 2025, doi: 10.1038/s41598-025-10075-1.
J. Liu, W. Sun, X. Zhao, J. Zhao, and Z. Jiang, “Deep feature fusion classification network (DFFCNet): Towards accurate diagnosis of COVID-19 using chest X-rays images,” Biomed Signal Process Control, vol. 76, p. 103677, Jul. 2022, doi: 10.1016/j.bspc.2022.103677.
Z. Feng and M. M. Giblin, “DeepFusionNet for realtime classification in iotbased crossmedia art and design using multimodal deep learning,” Sci Rep, vol. 15, no. 1, p. 34935, Oct. 2025, doi: 10.1038/s41598-025-18665-9.
S. A. El_Rahman, “Multimodal biometric systems based on different fusion levels of ECG and fingerprint using different classifiers,” Soft comput, vol. 24, no. 16, pp. 12599–12632, Aug. 2020, doi: 10.1007/s00500-020-04700-6.
A. J. Prakash, K. K. Patro, S. Samantray, P. Pławiak, and M. Hammad, “A Deep Learning Technique for Biometric Authentication Using ECG Beat Template Matching,” Information, vol. 14, no. 2, p. 65, Jan. 2023, doi: 10.3390/info14020065.
V. Agrawal, M. Hazratifard, H. Elmiligi, and F. Gebali, “Electrocardiogram (ECG)-Based User Authentication Using Deep Learning Algorithms,” Diagnostics, vol. 13, no. 3, p. 439, Jan. 2023, doi: 10.3390/diagnostics13030439.
D. Belo, N. Bento, H. Silva, A. Fred, and H. Gamboa, “ECG Biometrics Using Deep Learning and Relative Score Threshold Classification,” Sensors, vol. 20, no. 15, p. 4078, Jul. 2020, doi: 10.3390/s20154078.
F. Saeed, M. Hussain, and H. A. Aboalsamh, “Automatic Fingerprint Classification Using Deep Learning Technology (DeepFKTNet),” Mathematics, vol. 10, no. 8, p. 1285, Apr. 2022, doi: 10.3390/math10081285.
A. Balti, A. Hamdi, S. Abid, M. M. Ben Khelifa, and M. Sayadi, “Enhanced fingerprint classification through modified PCA with SVD and invariant moments,” Front Artif Intell, vol. 7, Aug. 2024, doi: 10.3389/frai.2024.1433494.
S. Bakheet, S. Alsubai, A. Alqahtani, and A. Binbusayyis, “Robust Fingerprint Minutiae Extraction and Matching Based on Improved SIFT Features,” Applied Sciences, vol. 12, no. 12, p. 6122, Jun. 2022, doi: 10.3390/app12126122.
P. Melzi, R. Tolosana, and R. Vera-Rodriguez, “ECG Biometric Recognition: Review, System Proposal, and Benchmark Evaluation,” IEEE Access, vol. 11, pp. 15555–15566, 2023, doi: 10.1109/ACCESS.2023.3244651.
A. Ali, R. Khan, I. Ullah, A. D. Khan, and A. Munir, “Minutiae Based Automatic Fingerprint Recognition: Machine Learning Approaches,” in 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing, IEEE, Oct. 2015, pp. 1148–1153. doi: 10.1109/CIT/IUCC/DASC/PICOM.2015.171.
A. Alshardan et al., “Multimodal biometric identification: leveraging convolutional neural network (CNN) architectures and fusion techniques with fingerprint and finger vein data,” PeerJ Comput Sci, vol. 10, 2024, doi: 10.7717/PEERJ-CS.2440.
M. Rahul, S. K. Jha, M. I. Khan, A. Prakash, P. ram Pal, and V. Yadav, “Feature Level Fusion of Fingerprints and IRIS for the Enhancement of Biometric System,” Recent Advances in Electrical & Electronic Engineering (Formerly Recent Patents on Electrical & Electronic Engineering), vol. 17, May 2024, doi: 10.2174/0123520965293198240418063311.
Z. Zhu et al., “Deep multi-layer perceptron-based evolutionary algorithm for dynamic multiobjective optimization,” Complex and Intelligent Systems, vol. 8, no. 6, pp. 5249–5264, Dec. 2022, doi: 10.1007/s40747-022-00745-2.
H. Zhu, “Multi-layered perceptron and its applications in biotechnology,” Theoretical and Natural Science, vol. 20, no. 1, pp. 159–165, Dec. 2023, doi: 10.54254/2753-8818/20/20230753.
B. Lindemann, T. Müller, H. Vietz, N. Jazdi, and M. Weyrich, “A survey on long short-term memory networks for time series prediction,” in Procedia CIRP, Elsevier B.V., 2021, pp. 650–655. doi: 10.1016/j.procir.2021.03.088.
G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,” Artif Intell Rev, vol. 53, no. 8, pp. 5929–5955, Dec. 2020, doi: 10.1007/s10462-020-09838-1.
R. A. Reddy, S. Kurian, G. M. Ramadan, B. Sh. Z. Abood, and M. I. Habelalmateen, “Multimodal Biometrics Classification Using Long Short Term Memory with Sigmoid Functions,” in 2023 3rd International Conference on Mobile Networks and Wireless Communications (ICMNWC), IEEE, Dec. 2023, pp. 1–6. doi: 10.1109/ICMNWC60182.2023.10435921.
U. A. Bhatti, H. Tang, G. Wu, S. Marjan, and A. Hussain, “Deep Learning with Graph Convolutional Networks: An Overview and Latest Applications in Computational Intelligence,” International Journal of Intelligent Systems, vol. 2023, no. 1, Jan. 2023, doi: 10.1155/2023/8342104.
G. Su, H. Wang, Y. Zhang, W. Zhang, and X. Lin, “Simple and deep graph attention networks,” Knowl Based Syst, vol. 293, p. 111649, Jun. 2024, doi: 10.1016/j.knosys.2024.111649.
A. G. Vrahatis, K. Lazaros, and S. Kotsiantis, “Graph Attention Networks: A Comprehensive Review of Methods and Applications,” Future Internet, vol. 16, no. 9, p. 318, Sep. 2024, doi: 10.3390/fi16090318.
J. H. Cabot and E. G. Ross, “Evaluating prediction model performance,” Surgery (United States), vol. 174, no. 3, pp. 723–726, Sep. 2023, doi: 10.1016/j.surg.2023.05.023.
M. Owusu-Adjei, J. Ben Hayfron-Acquah, T. Frimpong, and G. Abdul-Salaam, “Imbalanced class distribution and performance evaluation metrics: A systematic review of prediction accuracy for determining model performance in healthcare systems,” PLOS Digital Health, vol. 2, no. 11 November, Nov. 2023, doi: 10.1371/journal.pdig.0000290.
O. M. Amin Ali, S. Wahhab Kareem, and A. S. Mohammed, “Evaluation of Electrocardiogram Signals Classification Using CNN, SVM, and LSTM Algorithm: A review,” in 2022 8th International Engineering Conference on Sustainable Technology and Development (IEC), IEEE, Feb. 2022, pp. 185–191. doi: 10.1109/IEC54822.2022.9807511.
D. Jyotishi and S. Dandapat, “An LSTM-Based Model for Person Identification Using ECG Signal,” IEEE Sens Lett, vol. 4, no. 8, pp. 1–4, Aug. 2020, doi: 10.1109/LSENS.2020.3012653.
B. H. Kim and J. Y. Pyun, “ECG identification for personal authentication using LSTM-based deep recurrent neural networks,” Sensors (Switzerland), vol. 20, no. 11, pp. 1–17, Jun. 2020, doi: 10.3390/s20113069.
C. Li et al., “Spatio-temporal MLP network for seizure prediction using EEG signals,” Measurement (Lond), vol. 206, Jan. 2023, doi: 10.1016/j.measurement.2022.112278.
S. P. Balamurugan, “A Comprehensive Study on MLP and CNN, and the Implementation of Multi-Class Image Classification using Deep CNN,” in Machine Learning and Deep Learning Techniques for Medical Science, CRC Press, 2022, pp. 1–25. doi: 10.1201/9781003217497-1.
F. F. Taliningsih et al., “Biometric Verification Based on ECG Signal using 1 Dimensional Convolutional Neural Network,” in 2022 1st International Conference on Information System & Information Technology (ICISIT), IEEE, Jul. 2022, pp. 204–209. doi: 10.1109/ICISIT54091.2022.9872891.
F. Saeed, M. Hussain, and H. A. Aboalsamh, “Automatic Fingerprint Classification Using Deep Learning Technology (DeepFKTNet),” Mathematics, vol. 10, no. 8, p. 1285, Apr. 2022, doi: 10.3390/math10081285.
J. Jang, S. Park, J.-K. Kim, J. An, and S. Jung, “CNN-based Two Step R Peak Detection Method: Combining Segmentation and Regression,” in 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), IEEE, Jul. 2022, pp. 1910–1914. doi: 10.1109/EMBC48229.2022.9871227.
B. Tomas, M. Grabovac, and K. Tomas, “Application of the R-peak detection algorithm for locating noise in ECG signals,” Biomed Signal Process Control, vol. 72, Feb. 2022, doi: 10.1016/j.bspc.2021.103316.
L. Wu, X. Xie, and Y. Wang, “ECG Enhancement and R-Peak Detection Based on Window Variability,” Healthcare, vol. 9, no. 2, p. 227, Feb. 2021, doi: 10.3390/healthcare9020227.
J. Lee and M. Shin, “Using beat score maps with successive segmentation for ECG classification without R-peak detection,” Biomed Signal Process Control, vol. 91, May 2024, doi: 10.1016/j.bspc.2024.105982.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Muhammad Abdhi Priyatama, Dodon Turianto Nugrahadi, Irwan Budiman, Andi Farmadi, Mohammad Reza Faisal, Bedy Purnama, Puput Dani Prasetyo Adi, Luu Duc Ngo
This work is licensed under a Creative Commons Attribution 4.0 International License.
