Mapping Facial Expressions Based on Text for Virtual Counseling Chatbot Using IndoBERT Model
DOI:
https://doi.org/10.52436/1.jutif.2025.6.5.5039Keywords:
Chatbot, Early Marriage, Emotion Recognition, Facial Expression Mapping, IndoBERT, Virtual CounselingAbstract
Early marriage in Lombok remains a serious issue, with a prevalence rate of 16.59% in 2021, the second highest in Indonesia. Limited access to counseling services, especially in rural areas, poses a significant prevention challenge. This study developed a virtual counseling chatbot system capable of mapping text-based emotions to facial expressions to improve the effectiveness of counseling for early marriage prevention. The methodology involved training an IndoBERT model on a synthetic dataset to analyze conversation texts. The model was designed to classify user input into five functional emotion categories: Enthusiasm, Gentleness, Analytical, Inspirational, and Cautionary. Performance evaluation revealed that the IndoBERT model achieved an outstanding accuracy of 94% in its final phase. This result significantly surpassed other models evaluated, such as CNN (71.6%) and KNN (79%), confirming the superiority of the chosen approach The study concludes that the high-accuracy IndoBERT model is a robust foundation for empathetic virtual agents. This research provides a significant contribution to the fields of Affective Computing and Human-Computer Interaction by demonstrating an effective framework for mapping nuanced, functional emotions from Indonesian text to facial expressions. The proposed system not only offers a scalable technological solution for mental health challenges like early marriage prevention but also highlights the impact of advanced, context-aware NLP models in creating more human-like and empathetic user interactions.
Downloads
References
M. D. H. Rahiem, “COVID-19 and the surge of child marriages: A phenomenon in Nusa Tenggara Barat, Indonesia,” Child Abuse Negl, vol. 118, p. 105168, Aug. 2021, doi: 10.1016/j.chiabu.2021.105168.
F. A. Purnami, D. M. Maula, A. A. Nisa, R. Cahyaningtyas, A. R. Jundan, and F. Fitri, “Sosialisasi Dampak Pernikahan Dini dan Penguatan Mental Remaja sebagai Strategi Pencegahan Pernikahan Dini,” Welfare : Jurnal Pengabdian Masyarakat, vol. 2, no. 4, pp. 698–703, Dec. 2024, doi: 10.30762/welfare.v2i4.1979.
A. Sampurna, H. J. Ritonga, and A. R. Matondang, “Integration of Media Literacy in Religious Counseling for Preventing Early Marriage in Nias Barat,” International Journal of Islamic Education, Research and Multiculturalism (IJIERM), vol. 6, no. 3, pp. 1205–1218, Dec. 2024, doi: 10.47006/ijierm.v6i3.392.
L. Munira, P. Liamputtong, and P. Viwattanakulvanid, “Barriers and facilitators to access mental health services among people with mental disorders in Indonesia: A qualitative study,” Belitung Nurs J, vol. 9, no. 2, pp. 110–117, Apr. 2023, doi: 10.33546/bnj.2521.
C. D. Putra, R. A. S. Prayoga, M. Cinthya, R. Basatha, M. S. Akbar, and E. A. Elfaiz, “Mengembangkan Chatbot Empatik untuk Dukungan Kesehatan Mental: Solusi Inovatif dalam Pendampingan Psikologis,” Jurnal Ilmu Komputer dan Multimedia, vol. 1, no. 2, pp. 7–12, Dec. 2024, doi: 10.46510/ilkomedia.v1i2.19.
M. Laymouna, Y. Ma, D. Lessard, T. Schuster, K. Engler, and B. Lebouché, “Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review,” J Med Internet Res, vol. 26, p. e56930, Jul. 2024, doi: 10.2196/56930.
I. A. Assegaf et al., “PENGEMBANGAN CHATBOT KONSULTASI KESEHATAN MENTAL KESEHATAN MENTAL BERBASIS OPEN AI MODEL GPT-3.5 TURBO MENGGUNAKAN MEDIA WHATSAPP,” Jurnal Informatika Teknologi dan Sains (Jinteks), vol. 6, no. 4, pp. 785–793, Nov. 2024, doi: 10.51401/jinteks.v6i4.4749.
G. Park, J. Chung, and S. Lee, “Effect of AI chatbot emotional disclosure on user satisfaction and reuse intention for mental health counseling: a serial mediation model,” Current Psychology, vol. 42, no. 32, pp. 28663–28673, Nov. 2023, doi: 10.1007/s12144-022-03932-z.
H. Chin et al., “The Potential of Chatbots for Emotional Support and Promoting Mental Well-Being in Different Cultures: Mixed Methods Study,” J Med Internet Res, vol. 25, p. e51712, Oct. 2023, doi: 10.2196/51712.
J. Sun, “Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions,” Highlights in Science, Engineering and Technology, vol. 85, pp. 293–297, Mar. 2024, doi: 10.54097/agvjvq19.
Y. Zhang, “Computer-Assisted Human-Computer Interaction in Visual Communication,” Comput Aided Des Appl, vol. 18, no. S1, pp. 109–119, May 2020, doi: 10.14733/cadaps.2021.S1.109-119.
L. D. Cahya, A. Luthfiart, J. I. T. Krisna, S. Winarno, and A. Nugraha, “Improving Multi-label Classification Performance on Imbalanced Datasets Through SMOTE Technique and Data Augmentation Using IndoBERT Model,” Jurnal Nasional Teknologi dan Sistem Informasi, vol. 9, no. 3, pp. 290–298, Jan. 2024, doi: 10.25077/TEKNOSI.v9i3.2023.290-298.
W. M. Baihaqi and A. Munandar, “Sentiment Analysis of Student Comment on the College Performance Evaluation Questionnaire Using Naïve Bayes and IndoBERT,” JUITA : Jurnal Informatika, vol. 11, no. 2, p. 213, Nov. 2023, doi: 10.30595/juita.v11i2.17336.
M. Y. Baihaqi, E. Halawa, R. A. S. Syah, A. Nurrahma, and W. Wijaya, “Emotion Classification in Indonesian Language: A CNN Approach with Hyperband Tuning,” Jurnal Buana Informatika, vol. 14, no. 02, pp. 137–146, Oct. 2023, doi: 10.24002/jbi.v14i02.7558.
A. Zamsuri, S. Defit, and G. W. Nurcahyo, “Classification of Multiple Emotions in Indonesian Text Using The K-Nearest Neighbor Method,” Journal of Applied Engineering and Technological Science (JAETS), vol. 4, no. 2, pp. 1012–1021, Jun. 2023, doi: 10.37385/jaets.v4i2.1964.
S. Elysia and Herianto, “Chatbot Berbasis Retrieval Augmented Generation (RAG) untuk Peningkatan Layanan Informasi Sekolah,” Journal TIFDA (Technology Information and Data Analytic), vol. 1, no. 2, pp. 52–58, Dec. 2024, doi: 10.70491/tifda.v1i2.52.
F. R. Fatonah, D. S. Maylawati, and E. Nurlatifah, “Chatbot Edukasi Pra-Nikah berbasis Telegram Menggunakan Bidirectional Encoder Representations From Transformers (BERT),” Jurnal Algoritma, vol. 21, no. 2, pp. 29–40, Nov. 2024, doi: 10.33364/algoritma/v.21-2.1657.
A. Ardhiyansyah, C. Bakker, and S. G. Sijabat, “Dampak Teknologi Digital terhadap Kesejahteraan Mental: Tinjauan Interaksi, Tantangan, dan Solusi,” Jurnal Psikologi dan Konseling West Science, vol. 1, no. 04, pp. 181–188, Sep. 2023, doi: 10.58812/jpkws.v1i04.651.
M. Nesca, A. Katz, C. K. Leung, and L. M. Lix, “A scoping review of preprocessing methods for unstructured text data to assess data quality.,” Int J Popul Data Sci, vol. 7, no. 1, p. 1757, 2022, doi: 10.23889/ijpds.v6i1.1757.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 28th International Conference on Computational Linguistics, Stroudsburg, PA, USA: International Committee on Computational Linguistics, 2020, pp. 757–770. doi: 10.18653/v1/2020.coling-main.66.
S. William, Kenny, and A. Chowanda, “EMOTION RECOGNITION INDONESIAN LANGUAGE FROM TWITTER USING INDOBERT AND BI-LSTM,” Communications in Mathematical Biology and Neuroscience, vol. 2024, 2024, doi: 10.28919/cmbn/7858.
C. Shaw, P. LaCasse, and L. Champagne, “Exploring emotion classification of indonesian tweets using large scale transfer learning via IndoBERT,” Soc Netw Anal Min, vol. 15, no. 1, p. 22, Mar. 2025, doi: 10.1007/s13278-025-01439-6.
S. Sathyanarayanan, “Confusion Matrix-Based Performance Evaluation Metrics,” African Journal of Biomedical Research, pp. 4023–4031, Nov. 2024, doi: 10.53555/AJBR.v27i4S.4345.
R. J. L. Melatisudra, S. Utomo, S. Sutjiningtyas, and H. Hernawati, “Implementasi Pengenalan Ekspresi Wajah dengan Menggunakan Metode Convolutional Neural Network dan OpenCV Berbasis Webcam,” Journal of Computer System and Informatics (JoSYC), vol. 6, no. 1, pp. 339–348, Nov. 2024, doi: 10.47065/josyc.v6i1.6114.
K. Wisnudhanti and F. Candra, “Image Classification of Pandawa Figures Using Convolutional Neural Network on Raspberry Pi 4,” J Phys Conf Ser, vol. 1655, no. 1, p. 012103, Oct. 2020, doi: 10.1088/1742-6596/1655/1/012103.
D. M. Aprilla, F. Bimantoro, and I. G. P. S. Wijaya, “The Palmprint Recognition Using Xception, VGG16, ResNet50, MobileNet, and EfficientNetB0 Architecture,” JURNAL MEDIA INFORMATIKA BUDIDARMA, vol. 8, no. 2, p. 1065, Apr. 2024, doi: 10.30865/mib.v8i2.7577.
J. Terven, D.-M. Cordova-Esparza, J.-A. Romero-González, A. Ramírez-Pedraza, and E. A. Chávez-Urbiola, “A comprehensive survey of loss functions and metrics in deep learning,” Artif Intell Rev, vol. 58, no. 7, p. 195, Apr. 2025, doi: 10.1007/s10462-025-11198-7.
S. Farhadpour, T. A. Warner, and A. E. Maxwell, “Selecting and Interpreting Multiclass Loss and Accuracy Assessment Metrics for Classifications with Class Imbalance: Guidance and Best Practices,” Remote Sens (Basel), vol. 16, no. 3, p. 533, Jan. 2024, doi: 10.3390/rs16030533.
Rajesh Ediga, “Understanding the Technical Foundations of Large Language Models: Architectures, Training, and Applications,” Journal of Computer Science and Technology Studies, vol. 7, no. 7, pp. 154–161, Jul. 2025, doi: 10.32996/jcsts.2025.7.7.13.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Rifki Padilah, Rifki Wijaya, Shaufiah
This work is licensed under a Creative Commons Attribution 4.0 International License.
