Cross-Temporal Generalization of IndoBERT for Indonesian Hoax News Classification
DOI:
https://doi.org/10.52436/1.jutif.2025.6.5.4757Keywords:
Cross-set, Hoax Detection, IndoBERT, Model Generalization, Temporal Distribution ShiftAbstract
The spread of hoaxes in digital media poses a major challenge for automated detection systems as language and topics evolve over time. Although Transformer-based models such as IndoBERT have demonstrated high accuracy in previous studies, their performance across different time periods remains underexplored. This study examines the cross-temporal generalization ability of IndoBERT for hoax news classification. The model was trained on labeled articles from 2018–2023 and tested on data from 2025 to evaluate its robustness against temporal distribution shifts. The results indicate high accuracy on similar-period data (99.67–99.89%) but a decrease on 2025 data (95.45–95.87%), with most errors occurring as false negatives in the hoax class. These findings highlight the impact of temporal distribution shifts on model reliability and underscore the importance of adaptive strategies such as periodic retraining and domain-based data augmentation. Practically, this model has the potential to assist social media platforms and government institutions in developing dynamic and time-adaptive hoax detection systems. The cross-temporal approach employed in this study also offers methodological innovation compared to conventional random validation, as it better reflects real-world conditions where misinformation patterns continually evolve.
Downloads
References
E. Aïmeur, S. Amri, and G. Brassard, “Fake news, disinformation and misinformation in social media: a review,” Soc. Netw. Anal. Min., vol. 13, no. 1, p. 30, 2023, doi: 10.1007/s13278-023-01028-5.
E. Denniss and R. Lindberg, “Social media and the spread of misinformation: infectious and a threat to public health,” Health Promot. Int., vol. 40, no. 2, 2025, doi: 10.1093/heapro/daaf023.
Maria D Molina, S. Shyam Sundar, Thai Le, and Dongwon Lee, “‘Fake News’ Is Not Simply False Information: A Concept Explication and Taxonomy of Online Content,” Am. Behav. Sci., vol. 65, no. 2, pp. 180–212, Oct. 2019, doi: 10.1177/0002764219878224.
C. Zhou, K. Li, and Y. Lu, “Linguistic characteristics and the dissemination of misinformation in social media: The moderating effect of information richness,” Inf. Process. Manag., vol. 58, no. 6, p. 102679, 2021, doi: 10.1016/j.ipm.2021.102679.
S. Chen, L. Xiao, and A. Kumar, “Spread of misinformation on social media: What contributes to it and how to combat it,” Comput. Human Behav., vol. 141, p. 107643, 2023, doi: https://doi.org/10.1016/j.chb.2022.107643.
A. Wibawa, H. Joko Utomo, A. Bagus, and I. Dwi Arianto, “Political Hoaxes on Social Media Before the 2024 Regional Elections,” SHS Web Conf., vol. 212, no. August 2018, p. 04047, 2025, doi: 10.1051/shsconf/202521204047.
S. Loomba, A. de Figueiredo, S. J. Piatek, K. de Graaf, and H. J. Larson, “Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA,” Nat. Hum. Behav., vol. 5, no. 3, pp. 337–348, 2021, doi: 10.1038/s41562-021-01056-1.
S. Torpan et al., “Handling false information in emergency management: A cross-national comparative study of European practices,” Int. J. Disaster Risk Reduct., vol. 57, p. 102151, 2021, doi: 10.1016/j.ijdrr.2021.102151.
S. Muhammed T and S. K. Mathew, “The disaster of misinformation: a review of research in social media.,” Int. J. data Sci. Anal., vol. 13, no. 4, pp. 271–285, 2022, doi: 10.1007/s41060-022-00311-6.
K. D. Patel, “Fake News Detection on Natural Language Processing A Survey,” Int. J. Comput. Sci. Eng., vol. 7, no. 9, pp. 115–121, 2020, doi: 10.26438/ijcse/v7i9.115121.
Valentina Porcu, “Past vs. Present: Key Differences Between Conventional Machine Learning and Transformer Architectures,” Adv. Nonlinear Var. Inequalities, vol. 28, no. 2s, pp. 244–262, 2024, doi: 10.52783/anvi.v28.2537.
G. Tucudean, M. Bucos, B. Dragulescu, and C. D. Caleanu, “Natural language processing with transformers: a review,” PeerJ Comput. Sci., vol. 10, pp. 1–22, 2024, doi: 10.7717/PEERJ-CS.2222.
A. Fatwanto, F. Zamakhsyari, and R. Ndungi, “A Systematic Literature Review of BERT-based Models for Natural Language Processing Tasks,” J. Infotel, vol. 16, no. 4, pp. 713–728, 2024, doi: 10.20895/infotel.v16i4.1206.
R. Anggrainingsih, G. M. Hassan, and A. Datta, “Evaluating BERT-based language models for detecting misinformation,” Neural Comput. Appl., vol. 37, no. 16, pp. 9937–9968, 2025, doi: 10.1007/s00521-025-11101-z.
R. K. Kaliyar, A. Goswami, and P. Narang, “FakeBERT: Fake news detection in social media with a BERT-based deep learning approach,” Multimed. Tools Appl., vol. 80, no. 8, pp. 11765–11788, Mar. 2021, doi: 10.1007/s11042-020-10183-2.
J. Fawaid, A. Awalina, R. Y. Krisnabayu, and N. Yudistira, “Indonesia’s Fake News Detection using Transformer Network,” in Proceedings of the 6th International Conference on Sustainable Information Engineering and Technology, 2021, pp. 247–251. doi: 10.1145/3479645.3479666.
Muhammad Ikram Kaer Sinapoy, Yuliant Sibaroni, and Sri Suryani Prasetyowati, “Comparison of LSTM and IndoBERT Method in Identifying Hoax on Twitter,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 7, no. 3, pp. 657–662, 2023, doi: 10.29207/resti.v7i3.4830.
L. H. Suadaa, I. Santoso, and A. T. B. Panjaitan, “Transfer Learning of Pre-trained Transformers for Covid-19 Hoax Detection in Indonesian Language,” IJCCS (Indonesian J. Comput. Cybern. Syst., vol. 15, no. 3, p. 317, 2021, doi: 10.22146/ijccs.66205.
W. Ansar and S. Goswami, “Combating the menace: A survey on characterization and detection of fake news from a data science perspective,” Int. J. Inf. Manag. Data Insights, vol. 1, no. 2, p. 100052, 2021, doi: 10.1016/j.jjimei.2021.100052.
M. A. Hashmani, S. M. Jameel, M. Rehman, and A. Inoue, “Concept Drift Evolution In Machine Learning Approaches: A Systematic Literature Review,” Int. J. Smart Sens. Intell. Syst., vol. 13, no. 1, pp. 1–16, 2020, doi: 10.21307/ijssis-2020-029.
T. A. Chang et al., “Characterizing and Measuring Linguistic Dataset Drift,” Proc. Annu. Meet. Assoc. Comput. Linguist., vol. 1, pp. 8953–8967, 2023, doi: 10.18653/v1/2023.acl-long.498.
H. Wang, G. Luo, and R. Li, “A Short Text Classification Method Based on Combining Label Information and Self-attention Graph Convolutional Neural Network,” Commun. Comput. Inf. Sci., vol. 1330 CCIS, no. 1, pp. 670–677, 2021, doi: 10.1007/978-981-16-2540-4_50.
Z. Zhao, G. Chrysostomou, K. Bontcheva, and N. Aletras, “On the Impact of Temporal Concept Drift on Model Explanations,” Find. Assoc. Comput. Linguist. EMNLP 2022, pp. 4068–4083, 2022, doi: 10.18653/v1/2022.findings-emnlp.298.
I. Chalkidis and A. Søgaard, “Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting,” Proc. Annu. Meet. Assoc. Comput. Linguist., pp. 2441–2454, 2022, doi: 10.18653/v1/2022.findings-acl.192.
E. Shem-Tov, M. Sipper, and A. Elyasaf, “BERT Mutation: Deep Transformer Model for Masked Uniform Mutation in Genetic Programming,” Mathematics, vol. 13, no. 5, pp. 1–17, 2025, doi: 10.3390/math13050779.
A. Rogers, O. Kovaleva, and A. Rumshisky, “A primer in bertology: What we know about how bert works,” Trans. Assoc. Comput. Linguist., vol. 8, pp. 842–866, 2020, doi: 10.1162/tacl_a_00349.
M. H. Y. Binhammad, A. Othman, L. Abuljadayel, H. Al Mheiri, M. Alkaabi, and M. Almarri, “Investigating Advanced Generative Dialogue Systems for Educational Chatbots,” Creat. Educ., vol. 15, no. 08, pp. 1593–1626, 2024, doi: 10.4236/ce.2024.158096.
E. Cesario, C. Comito, and E. Zumpano, “A survey of the recent trends in deep learning for literature based discovery in the biomedical domain,” Neurocomputing, vol. 568, no. November 2023, 2024, doi: 10.1016/j.neucom.2023.127079.
N. J. Prottasha et al., “Transfer Learning for Sentiment Analysis Using BERT Based Supervised Fine-Tuning,” Sensors, vol. 22, no. 11, pp. 1–19, 2022, doi: 10.3390/s22114157.
L. M. Pham and H. C. The, “LNLF-BERT: Transformer for Long Document Classification with Multiple Attention Levels,” IEEE Access, vol. 12, no. November, pp. 165348–165358, 2024, doi: 10.1109/ACCESS.2024.3492102.
C. Toraman, E. H. Yilmaz, F. Aahi˙nuç, and O. Ozcelik, “Impact of Tokenization on Language Models: An Analysis for Turkish,” ACM Trans. Asian Low-Resource Lang. Inf. Process., vol. 22, no. 4, 2023, doi: 10.1145/3578707.
L. Stankevičius and M. Lukoševičius, “Extracting Sentence Embeddings from Pretrained Transformer Models,” Applied Sciences, vol. 14, no. 19. 2024. doi: 10.3390/app14198887.
X. Chen, P. Cong, and S. Lv, “A Long-Text Classification Method of Chinese News Based on BERT and CNN,” IEEE Access, vol. 10, pp. 34046–34057, 2022, doi: 10.1109/ACCESS.2022.3162614.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” COLING 2020 - 28th Int. Conf. Comput. Linguist. Proc. Conf., pp. 757–770, 2020, doi: 10.18653/v1/2020.coling-main.66.
R. Bagestra, A. Misbullah, Z. Zulfan, R. Rasudin, L. Farsiah, and S. Azizah, “Performance Assessment of Machine Learning and Transformer Models for Indonesian Multi-Label Hate Speech Detection,” vol. 2, no. 2, pp. 62–71, 2024, doi: 10.60084/ijds.v2i2.235.
G. Blanchard, A. A. Deshmukh, Ü. Dogan, G. Lee, and C. Scott, “Domain generalization by marginal transfer learning,” J. Mach. Learn. Res., vol. 22, no. 1, Jan. 2021, doi: 10.5555/3546258.3546260.
X. Zhou, Y. Jiang, and M. Bansal, “Data Factors for Better Compositional Generalization,” EMNLP 2023 - 2023 Conf. Empir. Methods Nat. Lang. Process. Proc., pp. 14549–14566, 2023, doi: 10.18653/v1/2023.emnlp-main.898.
E. Ben-Zaken, S. Ravfogel, and Y. Goldberg, “BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models,” Proc. Annu. Meet. Assoc. Comput. Linguist., vol. 2, pp. 1–9, 2022, doi: 10.18653/v1/2022.acl-short.1.
D. Rustam et al., “Development of Classification Method for Lecturer Area of Expertise Based on Scientific Publication Using BERT,” Int. J. Adv. Sci. Eng. Inf. Technol., vol. 14, no. 3, pp. 894–905, 2024, doi: 10.18517/ijaseit.14.3.19893.
G. Phillips et al., “Setting nutrient boundaries to protect aquatic communities: The importance of comparing observed and predicted classifications using measures derived from a confusion matrix,” Sci. Total Environ., vol. 912, no. November 2023, 2024, doi: 10.1016/j.scitotenv.2023.168872.
H. Shen, H. Jin, Á. A. Cabrera, A. Perer, H. Zhu, and J. I. Hong, “Designing Alternative Representations of Confusion Matrices to Support Non-Expert Public Understanding of Algorithm Performance,” Proc. ACM Human-Computer Interact., vol. 4, no. CSCW2, 2020, doi: 10.1145/3415224.
I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The Long-Document Transformer,” 2020, doi: https://doi.org/10.48550/arXiv.2004.05150.
Y. Hu, J. Ding, Z. Dou, and H. Chang, “Short-Text Classification Detector: A Bert-Based Mental Approach,” Comput. Intell. Neurosci., vol. 2022, 2022, doi: 10.1155/2022/8660828.
O. Chew, H.-T. Lin, K.-W. Chang, and K.-H. Huang, “Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis,” in Findings of the Association for Computational Linguistics: EACL 2024, Mar. 2024, pp. 1013–1025. doi: https://doi.org/10.48550/arXiv.2305.13654.
M. G. Kim, M. Kim, J. H. Kim, and K. Kim, “Fine-Tuning BERT Models to Classify Misinformation on Garlic and COVID-19 on Twitter,” Int. J. Environ. Res. Public Health, vol. 19, no. 9, 2022, doi: 10.3390/ijerph19095126.
P.-J. Lin, R. Balasubramanian, F. Liu, and N. Kandpal, “Efficient Model Development through Fine-tuning Transfer,” pp. 1–21, 2025, doi: https://doi.org/10.48550/arXiv.2503.20110.
M. Bucos and G. Țucudean, “Text Data Augmentation Techniques for Fake News Detection in the Romanian Language,” Applied Sciences, vol. 13, no. 13. 2023. doi: 10.3390/app13137389.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Agus Teguh Riadi, Fatma Indriani, Muhammad Itqan Mazdadi, Mohammad Reza Faisal, Rudi Herteno
This work is licensed under a Creative Commons Attribution 4.0 International License.
