Indonesian Hate Speech Detection: A Cross-Validated Benchmark of Machine Learning and Pre-trained Transformer Models with Statistical Significance Analysis
DOI:
https://doi.org/10.52436/1.jutif.2026.7.4.5176Keywords:
Cross-Validated Benchmark, Hate Speech Detection, Indonesian Social Media, Statistical SignificanceAbstract
Automated hate speech detection in Indonesian social media remains a persistent challenge due to dataset fragmentation, heterogeneous annotation schemes, and the lack of reproducible cross-model benchmarks with formal statistical validation. This study presents a cross-validated benchmark that systematically evaluates seven models — one majority baseline, three classical machine learning models (Logistic Regression, SVM, Random Forest), and three transformer-based pre-trained language models (DistilBERT, IndoRoBERTa, IndoBERT) — on a standardized multi-source Indonesian hate speech corpus comprising 14,043 samples from Twitter and Instagram. All models were trained and evaluated under identical conditions using stratified three-way splits (70/15/15) replicated across three random seeds, reporting mean ± standard deviation for F1-macro, precision-macro, recall-macro, and ROC-AUC. Paired t-tests and Cohen's d effect size analysis were applied to formally assess statistical significance and practical magnitude of performance differences. Results show that all transformer-based models significantly outperformed all classical ML models, with IndoBERT achieving the highest mean F1-macro of 0.8834 (±0.0026) and the lowest cross-seed variance among all models. Notably, IndoBERT and IndoRoBERTa were found to be statistically equivalent (p=0.7273, d=0.283), indicating that neither model is definitively superior for this task. Among classical models, SVM attained the best F1-macro of 0.8509. These findings confirm that domain-specific pre-training on Indonesian corpora contributes to both higher performance and superior cross-seed stability. The proposed benchmarking framework, standardized corpus, and statistical evaluation protocol provide a reproducible reference for future Indonesian hate speech detection research, thereby advancing the methodological standards of automated text classification in computer science and informatics, particularly for under-resourced language NLP research.
Downloads
References
H. Margono, M. Saud, and A. Ashfaq, “Dynamics of hate speech in social media: insights from Indonesia,” Glob. Knowl. Mem. Commun., Aug. 2024, doi: 10.1108/GKMC-11-2023-0464.
E. W. Pamungkas, D. G. P. Putri, and A. Fatmawati, “Hate Speech Detection in Bahasa Indonesia: Challenges and Opportunities,” Int. J. Adv. Comput. Sci. Appl., vol. 14, no. 6, 2023, doi: 10.14569/IJACSA.2023.01406125.
I. Tahir and M. G. F. Ramadhan, “Hate Speech On Social Media: Indonesian Netizens’ Hate Comments Of Presidential Talk Shows On Youtube,” LLT J. J. Lang. Lang. Teach., vol. 27, no. 1, pp. 230–251, Apr. 2024, doi: 10.24071/llt.v27i1.8180.
E. W. Pamungkas and P. Chiril, “Ngalawan Ujaran Sengit: hate speech detection in indonesian code-mixed social media data,” Lang. Resour. Eval., vol. 59, no. 3, pp. 2387–2414, Sep. 2025, doi: 10.1007/s10579-025-09810-x.
M. O. Ibrohim and I. Budi, “Hate speech and abusive language detection in Indonesian social media: Progress and challenges,” Heliyon, vol. 9, no. 8, p. e18647, Aug. 2023, doi: 10.1016/j.heliyon.2023.e18647.
P. A. Mufva, K. H. Chandra, K. F. Aji, I. A. Iswanto, and S. Joddy, “Performance comparison of deep learning approaches for Indonesian twitter hate speech detection using IndoBERTweet embedding,” Procedia Comput. Sci., vol. 269, pp. 1663–1671, 2025, doi: 10.1016/j.procs.2025.09.109.
I. Alfina, R. Mulia, M. I. Fanany, and Y. Ekanata, “Hate speech detection in the Indonesian language: A dataset and preliminary study,” in 2017 International Conference on Advanced Computer Science and Information Systems (ICACSIS), Bali: IEEE, Oct. 2017, pp. 233–238. doi: 10.1109/ICACSIS.2017.8355039.
M. O. Ibrohim and I. Budi, “Multi-label Hate Speech and Abusive Language Detection in Indonesian Twitter,” in Proceedings of the Third Workshop on Abusive Language Online, Florence, Italy: Association for Computational Linguistics, 2019, pp. 46–57. doi: 10.18653/v1/W19-3506.
L. Susanto et al., “A Multi-Labeled Dataset for Indonesian Discourse: Examining Toxicity, Polarization, and Demographics Information,” 2025, arXiv. doi: 10.48550/ARXIV.2503.00417.
M. A. Paz, J. Montero-Díaz, and A. Moreno-Delgado, “Hate Speech: A Systematized Review,” Sage Open, vol. 10, no. 4, p. 2158244020973022, Oct. 2020, doi: 10.1177/2158244020973022.
A. Tontodimamma, E. Nissi, A. Sarra, and L. Fontanella, “Thirty years of research into hate speech: topics of interest and their evolution,” Scientometrics, vol. 126, no. 1, pp. 157–179, Jan. 2021, doi: 10.1007/s11192-020-03737-6.
F. Alkomah and X. Ma, “A Literature Review of Textual Hate Speech Detection Methods and Datasets,” Information, vol. 13, no. 6, p. 273, May 2022, doi: 10.3390/info13060273.
M. Subramanian, V. Easwaramoorthy Sathiskumar, G. Deepalakshmi, J. Cho, and G. Manikandan, “A survey on hate speech detection and sentiment analysis using machine learning and deep learning models,” Alex. Eng. J., vol. 80, pp. 110–121, Oct. 2023, doi: 10.1016/j.aej.2023.08.038.
G. Ramos et al., “A comprehensive review on automatic hate speech detection in the age of the transformer,” Soc. Netw. Anal. Min., vol. 14, no. 1, p. 204, Oct. 2024, doi: 10.1007/s13278-024-01361-3.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proc. 28th Int. Conf. Comput. Linguistics (COLING), Barcelona, Spain (Online), 2020, pp. 757–770.
M. S. N. Raihan, N. Falih, M. P. Muslim, R. Zulfahmi, Jayanta, and I. Mardani, “Hoax and Hate Speech Detection in Indonesian Text Using IndoBERT and Explainable AI: A Systematic Literature Review,” in 2025 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS), Jakarta, Indonesia: IEEE, Dec. 2025, pp. 870–874. doi: 10.1109/ICIMCIS68501.2025.11327032.
B. Wilie et al., “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, Suzhou, China: Association for Computational Linguistics, 2020, pp. 843–857. doi: 10.18653/v1/2020.aacl-main.85.
B. Zaman, N. Humam, and I. K. Raharjana, “Large Language Model-Based Detoxification for Bahasa Indonesia,” Cybern. Inf. Technol., vol. 25, no. 3, pp. 3–21, Sep. 2025, doi: 10.2478/cait-2025-0019.
M. A. Lones, “How to avoid machine learning pitfalls: a guide for academic researchers,” 2021, doi: 10.48550/ARXIV.2108.02497.
N. Aliyah Salsabila, Y. Ardhito Winatmoko, A. Akbar Septiandri, and A. Jamal, “Colloquial Indonesian Lexicon,” in 2018 International Conference on Asian Language Processing (IALP), Bandung, Indonesia: IEEE, Nov. 2018, pp. 226–229. doi: 10.1109/IALP.2018.8629151.
V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” 2019, arXiv. doi: 10.48550/ARXIV.1910.01108.
J. Opitz, “A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice,” Trans. Assoc. Comput. Linguist., vol. 12, pp. 820–836, Jun. 2024, doi: 10.1162/tacl_a_00675.
D. Lakens, “Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs,” Front. Psychol., vol. 4, 2013, doi: 10.3389/fpsyg.2013.00863.
Y. Wahba, N. Madhavji, and J. Steinbacher, “A Comparison of SVM Against Pre-trained Language Models (PLMs) for Text Classification Tasks,” in Machine Learning, Optimization, and Data Science, vol. 13811, G. Nicosia, V. Ojha, E. La Malfa, G. La Malfa, P. Pardalos, G. Di Fatta, G. Giuffrida, and R. Umeton, Eds., in Lecture Notes in Computer Science, vol. 13811. , Cham: Springer Nature Switzerland, 2023, pp. 304–313. doi: 10.1007/978-3-031-25891-6_23.
S. D. A. Putri, M. O. Ibrohim, and I. Budi, “Abusive Language and Hate Speech Detection for Indonesian-Local Language in Social Media Text,” in Recent Advances in Information and Communication Technology 2021, vol. 251, P. Meesad, Dr. S. Sodsee, W. Jitsakul, and S. Tangwannawit, Eds., in Lecture Notes in Networks and Systems, vol. 251. , Cham: Springer International Publishing, 2021, pp. 88–98. doi: 10.1007/978-3-030-79757-7_9.
S. Mukherjee and S. Das, “Application of Transformer-Based Language Models to Detect Hate Speech in Social Media,” J. Comput. Cogn. Eng., vol. 2, no. 4, pp. 278–286, Dec. 2021, doi: 10.47852/bonviewJCCE2022010102.
X. Sika, D. Kisbianty, M. Istoningtyas, D. Z. Abidin, and A. N. Toscany, “Optimized RoBERTa–DeBERTa Ensemble for Multi-Class Sentiment Analysis on Highly Imbalanced Data,” J. Tek. Inform. Jutif, vol. 7, no. 2, pp. 1964–1980, Apr. 2026, doi: 10.52436/1.jutif.2026.7.2.5350.
E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” in Proc. 10th Int. Conf. Learn. Representations (ICLR), Virtual Event, 2022.
S. Nwaiwu, “Parameter-efficient fine-tuning for low-resource text classification: a comparative study of LoRA, IA3, and ReFT,” Front. Big Data, vol. 8, p. 1677331, Dec. 2025, doi: 10.3389/fdata.2025.1677331.
L. Xu, H. Xie, S. J. Qin, X. Tao, and F. L. Wang, “Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 48, no. 6, pp. 6107–6126, Jun. 2026, doi: 10.1109/TPAMI.2026.3657354.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Dodo Zaenal Abidin, Agus Siswanto, Chindra Saputra, Imelda Yose, Kaslin
This work is licensed under a Creative Commons Attribution 4.0 International License.
</a
