Mapping Gestures Based on Text Emotion Classification for a Virtual Chatbot for Early Marriage Consultation in Lombok Using RoBERTa Model
DOI:
https://doi.org/10.52436/1.jutif.2025.6.5.5038Keywords:
Early Marriage, Emotion Classification, Gesture Mapping, IndoRoBERTa, NLP, Virtual ChatbotAbstract
To address the persistent issue of early marriage among Indonesian adolescents, this study proposes a virtual counseling chatbot that classifies emotional cues in text using a fine-tuned IndoRoBERTa model. The emotion classification framework is designed to support counseling-based prevention efforts by moving beyond basic sentiment analysis and adopting five functional emotional categories such as ‘Enthusiastic’, ‘Gentle’, ‘Analytical’, ‘Inspirational’, and ‘Cautionary’ to align with psychological counseling styles. Built on fine-tuned IndoRoBERTa architecture, the model was trained in two phases: first with 2,500 manually validated samples yielding 92.8% accuracy, and then with 12,500 auto-labeled entries, resulting in 91.3% accuracy. Performance was assessed using accuracy, precision, recall, and F1-score. A gesture mapping layer was also integrated to enhance empathetic response generation. Each emotion label was paired with a predefined body gesture, grounded in counseling theory, to support future development of multimodal virtual agents capable of expressing emotions both textually and physically. The novelty lies in combining context-aware emotion classification with gesture mapping, enabling future development of expressive, culturally relevant, and empathetic virtual chatbot agents.
Downloads
References
R. Nabila, R. Roswiyani, and H. Satyadi, “A Literature Review of Factors Influencing Early Marriage Decisions in Indonesia,” in Proceedings of the 3rd Tarumanagara International Conference on the Applications of Social Sciences and Humanities (TICASH 2021), Atlantic Press, 2022, pp. 1392–1402. doi: 10.2991/assehr.k.220404.223.
D. Fadilah, “Tinjauan Dampak Pernikahan Dini dari Berbagai Aspek,” Pamator Journal, vol. 14, no. 2, pp. 88–94, Nov. 2021, doi: 10.21107/pamator.v14i2.10590.
Z. Ayudiputri, A. Nur, S. Amanda, and F. Hanifa, “Determinants of Child Marriage in Indonesia : A Systematic Review,” Journal of Community Medicine and Public Health Research, vol. 5, no. 2, pp. 216–227, Nov. 2024, doi: 10.20473/jcmphr.v5i2.45777.
L. Wang, D. Wang, F. Tian, Z. Peng, X. Fan, Z. Zhang et al., “CASS: Towards Building a Social-Support Chatbot for Online Health Community,” in Conference on Computer-Supported Cooperative Work & Social Computing (CSCW), Feb. 2021, pp. 1–31. doi: https://doi.org/10.48550/arXiv.2101.01583.
S. Khandelwal, “SOCIAL COMPANION CHATBOT FOR HUMAN COMMUNICATION USING ML AND NLP,” International Journal of Engineering Applied Sciences and Technology, vol. 8, pp. 321–324, 2023, doi: https://doi.org/10.33564/IJEAST.2023.v08i01.048.
R. E. Guingrich and M. S. A. Graziano, “Chatbots as Social Companions: How People Perceive Consciousness, Human Likeness, and Social Health Benefits in Machines,” in Oxford Intersections: AI in Society, Oxford University Press, 2025. doi: https://doi.org/10.1093/9780198945215.001.0001.
P. F. Muhammad, R. Kusumaningrum, and A. Wibowo, “Sentiment Analysis Using Word2vec and Long Short-Term Memory (LSTM) for Indonesian Hotel Reviews,” in Procedia Computer Science, Elsevier B.V., 2021, pp. 728–735. doi: 10.1016/j.procs.2021.01.061.
N. Hilmiaji, K. M. Lhaksmana, and M. D. Purbalaksono, “Identifying Emotion on Indonesian Tweets using Convolutional Neural Networks,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 5, no. 3, pp. 584–593, Jun. 2021, doi: 10.29207/resti.v5i3.3137.
N. G. Ramadhan, “Indonesian Online News Topics Classification using Word2Vec and K-Nearest Neighbor,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 5, no. 6, pp. 1083–1089, Dec. 2021, doi: 10.29207/resti.v5i6.3547.
M. I. K. Sinapoy, Y. Sibaroni, and S. S. Prasetyowati, “Comparison of LSTM and IndoBERT Method in Identifying Hoax on Twitter,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 7, no. 3, pp. 657–662, Jun. 2023, doi: 10.29207/resti.v7i3.4830.
S. William, Kenny, and A. Chowanda, “EMOTION RECOGNITION INDONESIAN LANGUAGE FROM TWITTER USING INDOBERT AND BI-LSTM,” Communications in Mathematical Biology and Neuroscience, vol. 2024, 2024, doi: 10.28919/cmbn/7858.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” ArXiv, vol. abs/1907.11692, Jul. 2019, doi: https://doi.org/10.48550/arXiv.1907.11692.
Y. O. Sihombing, R. F. Rachmadi, S. Sumpeno, and Moh. J. Mubarok, “Optimizing IndoRoBERTa Model for Multi-Class Classification of Sentiment & Emotion on Indonesian Twitter,” in Proceeding - IEEE 10th Information Technology International Seminar, ITIS 2024, Institute of Electrical and Electronics Engineers Inc., 2024, pp. 12–17. doi: 10.1109/ITIS64716.2024.10845566.
T. Widarmanti, M. P. Widodo, D. P. Ramadhani, and M. Danlami, “Text Emotion Detection: Discover the Meaning Behind YouTube Comments Using Indo RoBERTa,” in ICACNIS 2022 - 2022 International Conference on Advanced Creative Networks and Intelligent Systems: Blockchain Technology, Intelligent Systems, and the Applications for Human Life, Proceeding, Institute of Electrical and Electronics Engineers Inc., 2022, p. 1. doi: 10.1109/ICACNIS57039.2022.10055265.
F. M. Plaza-Del-Arco, A. Curry, A. C. Curry, and D. Hovy, “Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia: ELRA and ICCL, May 2024, pp. 5696–5710. doi: 10.48550/arXiv.2403.01222.
A. Koufakou and E. Nieves, “Review of recent emotion-annotated text corpora and resources,” Lang Resour Eval, pp. 1–35, Jun. 2025, doi: 10.1007/s10579-025-09828-1.
E. (Grace) Park, “I Trust You, but Let Me Talk to AI: The Role of the Chat Agents, Empathy, and Health Issues in Misinformation Guidance,” International Journal of Strategic Communication, vol. 19, no. 2, pp. 231–260, Mar. 2025, doi: 10.1080/1553118X.2025.2462087.
Y. Li, K. Li, H. Ning, X. Xia, Y. Guo, C. Wei et al., “Towards an Online Empathetic Chatbot with Emotion Causes,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA: ACM, Jul. 2021, pp. 2041–2045. doi: 10.1145/3404835.3463042.
H. Li, G. K. Rajbahadur, D. Lin, C.-P. Bezemer, Z. Ming, and Jiang, “Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting,” vol. 12, pp. 70676–70689, Jan. 2024, doi: 10.1109/ACCESS.2024.3402543.
S. Sathyanarayanan, “Confusion Matrix-Based Performance Evaluation Metrics,” African Journal of Biomedical Research, vol. 27, pp. 4023–4031, Nov. 2024, doi: 10.53555/AJBR.v27i4S.4345.
Y. Zhang, M. Safdar, J. Xie, J. Li, M. Sage, and Y. F. Zhao, “A systematic review on data of additive manufacturing for machine learning applications: the data quality, type, preprocessing, and management,” J Intell Manuf, vol. 34, no. 8, pp. 3305–3340, Dec. 2023, doi: 10.1007/s10845-022-02017-9.
Y. Li, X. Ren, F. Zhao, and S. Yang, “A Zeroth-Order Adaptive Learning Rate Method to Reduce Cost of Hyperparameter Tuning for Deep Learning,” Applied Sciences, vol. 11, no. 21, p. 10184, Oct. 2021, doi: 10.3390/app112110184.
J. S. Hwang, S. S. Lee, J. W. Gil, and C. K. Lee, “Determination of Optimal Batch Size of Deep Learning Models with Time Series Data,” Sustainability (Switzerland), vol. 16, no. 14, Jul. 2024, doi: 10.3390/su16145936.
S. Ahn, S. Kim, J. Ko, and S.-Y. Yun, “Fine tuning Pre trained Models for Robustness Under Noisy Labels,” in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, (IJCAI-24), International Joint Conferences on Artificial Intelligence Organization, Oct. 2023, pp. 3643–3651. doi: https://doi.org/10.48550/arXiv.2310.17668.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Nov. 2020, pp. 757–770. doi: https://doi.org/10.48550/arXiv.2011.00677.
W. Stigall, M. A. Al Hafiz Khan, D. Attota, F. Nweke, and Y. Pei, “Large Language Models Performance Comparison of Emotion and Sentiment Classification,” in Proceedings of the 2024 ACM Southeast Conference, ACMSE 2024, Association for Computing Machinery, Inc, Apr. 2024, pp. 60–68. doi: 10.1145/3603287.3651183.
J. Opitz, “A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice,” Transactions of the Association for Computational Linguistics 2024, vol. 12, pp. 820–836, Apr. 2024, doi: 10.1162/tacl_a_00675.
T. Schlosser, M. Friedrich, T. Meyer, D. Kowerko, and J. Professorship, A Consolidated Overview of Evaluation and Performance Metrics for Machine Learning and Computer Vision. 2024. doi: 10.13140/RG.2.2.14331.69928.
O. Rainio, J. Teuho, and R. Klén, “Evaluation metrics and statistical tests for machine learning,” Sci Rep, vol. 14, no. 1, p. 6086, Dec. 2024, doi: 10.1038/s41598-024-56706-x.
S. Jiang, J. Li, Y. Wang, B. Huang, Z. Zhang, and T. Xu, “Delving into Sample Loss Curve to Embrace Noisy and Imbalanced Data,” in Proceedings of the AAAI Conference on Artificial Intelligence, Association for the Advancement of Artificial Intelligence, Dec. 2021, pp. 7024–7032. doi: https://doi.org/10.48550/arXiv.2201.00849.
A. Diwan, R. Sunil, P. Mer, R. Mahadeva, and S. P. Patole, “Advancements in Emotion Classification via Facial and Body Gesture Analysis: A Survey,” Expert Syst, vol. 42, no. 2, p. e13759, 2025, doi: https://doi.org/10.1111/exsy.13759.
J. Hofmann, E. Troiano, K. Sassenberg, and R. Klinger, “Appraisal Theories for Emotion Classification in Text,” in Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain: International Committee on Computational Linguistics, Nov. 2020, pp. 125–138. doi: doi.org/10.48550/arXiv.2003.14155.
L. Zhang, J. Yu, S. Zhang, L. Li, Y. Zhong, G. Liang et al., “Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations,” CoRR, vol. abs/2406.15000, Jun. 2024, doi: 10.48550/arXiv.2406.15000.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Adam Zahran Ramadhan, Rifki Wijaya, Shaufiah
This work is licensed under a Creative Commons Attribution 4.0 International License.
