Comparative Analysis of Baseline IndoBERT, Class-Weighted IndoBERT, and SMOTE with Support Vector Machine for Handling Imbalanced Sentiment Classification in Indonesian

Riya Widayanti; Fitriana Cendra  Kasih

doi:10.52436/1.jutif.2026.7.3.5692

Authors

Riya Widayanti Department of Informatics, Faculty of Computer Science, Esa Unggul University, Jakarta, Indonesia
Fitriana Cendra Kasih Department of Informatics, Faculty of Computer Science, Esa Unggul University, Jakarta, Indonesia

DOI:

https://doi.org/10.52436/1.jutif.2026.7.3.5692

Keywords:

Class Weighting, Imbalanced Dataset, IndoBERT, SMOTE, Sentiment Classification, SVM

Abstract

Imbalanced data distribution is a common issue in Indonesian sentiment classification and significantly affects the performance of classification models. This study investigates three approaches, namely SMOTE combined with Support Vector Machine (SMOTE + SVM), Baseline IndoBERT, and Class-Weighted IndoBERT. The dataset consists of Google Maps reviews, which are categorized into positive, neutral, and negative sentiments. Prior to model training, the data undergo preprocessing steps including cleaning, normalization, and tokenization. Model performance is evaluated using confusion matrix analysis and macro-averaged F1-score. The results show that Baseline IndoBERT achieves a macro F1-score of 0.598, followed by Class-Weighted IndoBERT with 0.582, while SMOTE + SVM obtains the lowest performance at 0.545. Despite having slightly lower overall performance, Class-Weighted IndoBERT demonstrates a more balanced capability in recognizing minority classes. These findings indicate that incorporating class-weighting mechanisms into transformer-based models can help mitigate bias toward majority classes and improve minority class recognition. From a scientific perspective, this study provides empirical evidence on how imbalance-aware learning strategies influence the behavior of transformer-based models in imbalanced text classification tasks. Furthermore, this study highlights the importance of using macro-averaged evaluation metrics to ensure a more comprehensive and fair assessment of model performance, particularly in low-resource and imbalanced language settings.

Downloads

Download data is not yet available.

References

N. A. R. Putri and Ardiansyah, “Analisis Sentimen Terhadap Kemajuan Kecerdasan Buatan di Indonesia Menggunakan BERT dan RoBERTa,” J. Sains dan Inform., vol. 9, no. 2, pp. 136–145, 2023, doi: 10.34128/jsi.v9i2.649.

Y. Mao, Q. Liu, and Y. Zhang, “Sentiment analysis methods , applications , and challenges : A systematic literature review,” J. King Saud Univ. - Comput. Inf. Sci., vol. 36, no. 4, p. 102048, 2024, doi: 10.1016/j.jksuci.2024.102048.

H. Murfi, S. Theresia Gowandi, G. Ardaneswari, and S. Nurrohmah, “BERT-based combination of convolutional and recurrent neural network for indonesian sentiment analysis,” Appl. Soft Comput., vol. 151, pp. 1–15, 2024, doi: 10.1016/j.asoc.2023.111112.

R. C. Rivaldi and T. D. Wismarini, “Analisis Sentimen Pada Ulasan Produk Dengan Metode Natural Language Processing ( NLP ) ( Studi Kasus Zalika Store 88 Shopee ),” J. Ilm. Elektron. DAN Komput., vol. 17, no. 1, pp. 120–128, 2024, doi: https://doi.org/10.51903/elkom.v17i1.1680 JURNAL.

A. A. Purnama and Y. R. Sipayung, “Sentiment Analysis of Public Service Using Naïve Bayes Classifier,” J. Inf. Syst. Informatics, vol. 7, no. 3, pp. 2439–2457, 2025, doi: 10.51519/journalisi.v7i3.1207.

M. R. Tanjung, M. Iqbal, and Z. Sitorus, “Analisis Sentimen Google Review terhadap Mutu Kualitas Pendidikan pada Perguruan Tinggi STIE Al-Washliyah Sibolga dengan Metode Lexicon dan Algoritma Naive Bayes,” Jatilima J. Multimed. Dan Teknol. Inf., vol. 07, no. 02, pp. 400–412, 2025, doi: https://doi.org/10.54209/jatilima.v7i02.1549.

B. Atmadja, “Analisis Sentimen Bahasa Indonesia Pada Tempat Wisata di Kabupaten Sukabumi Dengan Naïve Bayes,” vol. 15, no. 2, pp. 371–382, 2022, doi: https://doi.org/10.51903/elkom.v15i2.872.

K. H. Prastiawan and D. Yuniarto, “Analisis Sentimen Publik terhadap Program Makan Bergizi Gratis dengan Algoritma Naive Bayes,” vol. 4, no. 4, pp. 5412–5419, 2025, doi: https://doi.org/10.31004/riggs.v4i4.3652.

G. Shini and S. V, “Performance Evaluation of Sentiment Analysis on Balanced and Imbalanced Dataset Using Ensemble Approach,” INDIAN J. Sci. Technol., vol. 15, no. 17, pp. 790–797, 2022, doi: htt ps://doi.org/ 10.17485/IJST/v15i17.2339.

P. A. Perwira and N. I. Widiastuti, “Imbalance Dataset in Aspect-Based Sentiment Analysis on Game Genshin Impact Review,” J. INFOTEL, vol. 16, no. 1, pp. 71–81, 2024, doi: 10.20895/INFOTEL.V16I1.984.

I. S. Ritonga, Wanayumini, and D. Hartama, “Sentiment Classification in Imbalanced Data : Trade-Offs Between Metrics and Real-World Relevanced,” J. Tek. Inform., vol. 18, no. 2, pp. 303–315, 2025, doi: https://doi.org/10.15408/jti.v18i2.46452.

F. Ayu, D. Aryanti, A. Luthfiarta, D. Adiwinata, and I. Soeroso, “Aspect-Based Sentiment Analysis with LDA and IndoBERT Algorithm on Mental Health App : Riliv,” J. Appl. Informatics Comput., vol. 9, no. 2, pp. 361–375, 2025, doi: https://doi.org/10.30871/jaic.v9i2.8958.

Sudianto, “PRE-TRAINED BERT ARCHITECTURE ANALYSIS FOR INDONESIAN QUESTION ANSWER MODEL,” J. Appl. Eng. Technol. Sci., vol. 6, no. 1, pp. 60–68, 2024, doi: https://doi.org/10.37385/jaets.v6i1.4746.

Vidya Chandradev, I Made Agus Dwi Suarjaya, and I Putu Agung Bayupati, “Analisis Sentimen Review Hotel Menggunakan Metode Deep Learning BERT,” J. Buana Inform., vol. 14, no. 02, pp. 107–116, 2023, doi: 10.24002/jbi.v14i02.7244.

E. Yulianti and N. K. Nissa, “ABSA of Indonesian customer reviews using IndoBERT: single-sentence and sentence-pair classification approaches,” Bull. Electr. Eng. Informatics, vol. 13, no. 5, pp. 3579–3589, 2024, doi: 10.11591/eei.v13i5.8032.

G. Medantoro and M. Muljono, “Comparative Analysis of IndoBERT and Classic Machine Learning Models for Sentiment Classification of Education Policy on Social Media X,” J. Appl. Informatics Comput., vol. 10, no. 1, pp. 548–557, 2026, doi: https://doi.org/10.30871/jaic.v10i1.11723.

Y. Setiawan and L. A. Wulandhari, “Comparative Analysis of IndoBERT and LSTM for Multi- Label Text Classification of Indonesian Motivation Letter,” JOIN (Jurnal Online Inform., vol. 10, no. 2, pp. 260–269, 2025, doi: 10.15575/join.v10i2.1499.

M. A. Fathin, Y. Sibaroni, and S. S. Prasetyowati, “Handling Imbalance Dataset on Hoax Indonesian Political News Classification using IndoBERT and Random Sampling,” J. Media Inform. Budidarma, vol. 8, no. 2021, pp. 352–360, 2024, doi: 10.30865/mib.v8i1.7099.

Y. A. Singgalen, “Performance Analysis of IndoBERT for Sentiment Classification in Indonesian Hotel Review Data,” J. Inf. Syst. Res., vol. 6, no. 2, pp. 978–988, 2025, doi: 10.47065/josh.v6i2.6505.

F. Pralienka, B. Muhamad, E. Mulyani, M. S. Bunga, and A. Farhan, “Class Balancing Methods Comparison for Software Requirements Classification on Support Vector Machines,” Sink. J. dan Penelit. Tek. Inform., vol. 7, no. 2, pp. 1196–1208, 2023, doi: https://doi.org/10.33395/sinkron.v8i2.12415 e-ISSN.

A. Fitri, D. Anggraeni, and I. M. Tirta, “Implementasi Random Forest Menggunakan SMOTE untuk Analisis Sentimen Ulasan Aplikasi Sister for Students UNEJ,” J. Nas. Teknol. dan Sist. Inf., vol. 02, no. 2022, pp. 163–172, 2023, doi: https://doi.org/10.25077/TEKNOSI.v9i2.2023.163-172.

I. N. Switrayana, D. Ashadi, H. Hairani, and A. Aminuddin, “Sentiment Analysis and Topic Modeling of Kitabisa Applications using Support Vector Machine ( SVM ) and Smote-Tomek Links Methods,” Int. J. Eng. Comput. Sci. Appl., vol. 2, no. 2, pp. 87–98, 2023, doi: 10.30812/IJECSA.v2i2.3406.

A. A. Qolbu, N. Fitriyati, and N. Inayah, “Performa Naïve Bayes, SVM, dan IndoBERT pada Analisis Sentimen Twitter IndiHome dengan Strategi Penanganan Data Tidak Seimbang,” J. FOURIER, vol. 814, no. 1, pp. 29–44, 2025, doi: 10.14421/fourier.2025.141.29-44.

A. B. Siva and L. Hoki, “Comparison of IndoBERT and SVM Performance in Sentiment Analysis of Digital Education Platforms,” Sink. J. dan Penelit. Tek. Inform., vol. 10, no. 1, pp. 64–74, 2026, doi: 10.33395/sinkron.v10i1.15472.

M. Cristina, H. Lee, J. Braet, and J. Springael, “Performance Metrics for Multilabel Emotion Classification : Comparing Micro , Macro , and Weighted F1-Scores,” Appl. Sci., vol. 14, no. 21, 2024, doi: 10.3390/app14219863.

R. Erama, “Pemanfaatan Platform Cloud Google Colab Untuk Scraping Komentar Tiktok Pada Konten Gorontalo sebagai Dasar Analisis Respons Warganet,” J. Appl. Eng. Sci., vol. 1, no. 2, pp. 124–134, 2025, doi: 10.65177/jaes.v1i2.38.

D. S. Utami, A. Erfina, and M. Id, “Analisis Sentimen Ulasan Terkait UNESCO Global Geopark Di Google Maps dengan Algoritma Naive Bayes,” J. Sains Komput. Inform., vol. 6, no. 2, pp. 1154–1170, 2022, doi: 10.30645/j-sakti.v6i2.524.

S. Khairunnisa, A. Adiwijaya, and S. Al Faraby, “Pengaruh Text Preprocessing terhadap Analisis Sentimen Komentar Masyarakat pada Media Sosial Twitter (Studi Kasus Pandemi COVID-19),” J. Media Inform. Budidarma, vol. 5, no. 2, p. 406, 2021, doi: 10.30865/mib.v5i2.2835.

A. Bijaksana and P. Negara, “The Influence Of Applying Stopword Removal And Smote On Indonesian Sentiment Classification,” LONTAR Komput. J. Ilm. TEKONOLOGI Inf., vol. 14, no. 3, pp. 172–185, 2023, doi: 10.24843/LKJITI.2023.v14.i03.p05.

A. Jazuli, Widowati, and R. Kusumaningrum, “Optimizing Aspect-Based Sentiment Analysis Using BERT for Comprehensive Analysis of Indonesian Student Feedback,” Appl. Sci., vol. 15, no. 1, pp. 1–28, 2025, doi: 10.3390/app15010172.

O. A. Irmawan, I. Budi, A. B. Santoso, and P. K. Putra, “Improving Sentiment Analysis and Topic Extraction in Indonesian Travel App Reviews Through BERT Fine-Tuning,” J. Nas. Pendidik. Tek. Inform., vol. 13, no. 2, pp. 359–370, 2024, doi: 10.23887/janapati.v13i2.77028.

P. Sayarizki and H. Nurrahmi, “Implementation of IndoBERT for Sentiment Analysis of Indonesian Presidential Candidates,” Indones. J. Comput., vol. 9, no. August, pp. 61–72, 2024, doi: 10.34818/indojc.2024.9.2.934.

H. Ma’rifah, A. P. Wibawa, and M. I. Akbar, “Klasifikasi Artikel Ilmiah Dengan Berbagai Skenario Preprocessing,” Sains, Apl. Komputasi dan Teknol. Inf., vol. 2, no. 2, p. 70, 2020, doi: 10.30872/jsakti.v2i2.2681.

I. Daqiqil, H. Saputra, Syamsudhuha, R. Kurniawan, and Y. Andriyani, “Sentiment analysis of student evaluation feedback using transformer-based language models,” Indones. J. Electr. Eng. Comput. Sci., vol. 36, no. 2, pp. 1127–1139, 2024, doi: 10.11591/ijeecs.v36.i2.pp1127-1139.

A. Jazuli, Widowati, and R. Kusumaningrum, “Aspect-based sentiment analysis on student reviews using the Indo-Bert base model,” E3S Web Conf., vol. 448, pp. 1–10, 2023, doi: 10.1051/e3sconf/202344802004.

N. Sholihah, F. F. Abdulloh, and M. Rahardi, “Sentiment Analysis on KPU Performance Post-2024 Election via YouTube Comments Using BERT,” Sink. J. dan Penelit. Tek. Inform., vol. 8, no. 4, pp. 2222–2232, 2024, doi: 10.33395/sinkron.v8i4.14040.

L. D. Cahya, A. Luthfiarta, J. Imanuel, S. Winarno, and A. Nugraha, “Improving Multi-label Classification Performance on Imbalanced Datasets Through SMOTE Technique and Data Augmentation Using IndoBERT Model,” J. Nas. Teknol. dan Sist. Inf., vol. 09, no. 03, pp. 290–298, 2023, doi: 10.25077/TEKNOSI.v9i3.2023.290-298.

A. Kumar, A. Murugappan, T. Esther, A. Murugappan, and T. Esther, “Imbalanced aspect categorization using bidirectional encoder Imbalanced aspect categorization bidirectional encoder representation from using transformers representation from transformers,” Procedia Comput. Sci., vol. 218, pp. 757–765, 2023, doi: 10.1016/j.procs.2023.01.056.

K. G. R. Narayan et al., “Attenuating majority attack class bias using hybrid deep learning based IDS framework,” J. Netw. Comput. Appl., vol. 230, p. 103954, 2024, doi: https://doi.org/10.1016/j.jnca.2024.103954.

M. N. Razali, N. Arbaiy, and P. Lin, “Optimizing Multiclass Classification Using Convolutional Neural Networks with Class Weights and Early Stopping for Imbalanced Datasets,” MDPI, vol. 14, no. 4, pp. 1–14, 2025, doi: 10.3390/electronics14040705.

I. Araf, A. Idri, and I. Chairi, “Cost-sensitive learning for imbalanced medical data: a review,” Artif. Intell. Rev., vol. 57, no. 4, p. 80, 2024, doi: 10.1007/s10462-023-10652-8.

A. S. Dina, A. B. Siddique, and D. Manivannan, “A deep learning approach for intrusion detection in Internet of Things using focal loss function,” Internet of Things, vol. 22, p. 100699, 2023, doi: https://doi.org/10.1016/j.iot.2023.100699.

D. A. Lestari, Y. Sibaroni, and S. S. Prasetiyowati, “Sentiment Analysis of Transportation Application Reviews with SVM on Handling Imbalanced Data Using SMOTE,” in 2025 International Conference on Data Science and Its Applications (ICoDSA), 2025, pp. 287–292. doi: 10.1109/ICoDSA67155.2025.11157024.

S. F. Taskiran, B. Turkoglu, E. Kaya, and T. Asuroglu, “A comprehensive evaluation of oversampling techniques for enhancing text classification performance,” Sci. Rep., vol. 15, no. 21631, pp. 1–20, 2025, doi: 10.1038/s41598-025-05791-7 1.

F. Sağlam and M. A. Cengiz, “A novel SMOTE-based resampling technique trough noise detection and the boosting procedure,” Expert Syst. Appl., vol. 200, p. 117023, 2022, doi: https://doi.org/10.1016/j.eswa.2022.117023.

G. Wang and M. M. Jaber, “A Deep Learning Approach to Sentiment Analysis of Hotel Reviews : Comparing BERT and LSTM Models,” Int. J. Adv. Artif. Intell. Mach. Learn., vol. 2, no. 2, pp. 67–75, 2025, doi: 10.58723/ijaaiml.v2i2.403.

S. Maulana, N. S. Fatonah, G. Firmansyah, A. M. Widodo, and U. E. Unggul, “PREDICTING TECHNICAL INTERN TRAINING PROGRAM TRAINEE SUCCESS : A COMPARATIVE MACHINE LEARNING,” J. INOVTEK POLBENG, vol. 10, no. 3, pp. 1753–1761, 2025, doi: https://doi.org/10.35314/1r93bf26.

D. Marutho and V. G. Utomo, “Benchmarking IndoBERT and Transformer Models for Sentiment Classification on Indonesian E-Government Service Reviews,” J. Transform., vol. 23, no. 1, pp. 85–95, 2025, doi: 10.26623/transformatika.v23i1.12095.

A. Ramadhan and U. Zaky, “Cross-Lingual Sentiment Analysis for Indonesian Monetary Policy,” J. Sci. Res. Educ. Technol., vol. 4, no. 4, pp. 2588–2601, 2025, doi: 10.58526/jsret.v4i4.943.