Comparing BERTBase, DistilBERT and RoBERTa in Sentiment Analysis for Disaster Response

Hafiz Budi Firmansyah; Aidil Afriansyah; Valerio Lorini

doi:10.52436/1.jutif.2025.6.5.4766

Authors

Hafiz Budi Firmansyah Department of Informatics, Institut Teknologi Sumatera, Lampung, Indonesia
Aidil Afriansyah Department of Informatics, Institut Teknologi Sumatera, Lampung, Indonesia
Valerio Lorini European Parliament, Luxembourg

DOI:

https://doi.org/10.52436/1.jutif.2025.6.5.4766

Keywords:

Deep Learning, Disaster Response, Sentiment Analysis, Social Media

Abstract

Social media platforms are vital for real-time communication during disasters, providing insights into public emotions and urgent needs. This study evaluates the performance of three transformer-based models—BERTBase, DistilBERT, and RoBERTa—for sentiment analysis on disaster-related social media data. Using a multilingual dataset sourced from the Social Media for Disaster Risk Management (SMDRM) platform, the models were assessed on classification metrics including accuracy, precision, recall, and weighted F1-score. The results show that RoBERTa consistently outperforms the others in classification performance, while DistilBERT offers superior computational efficiency. The analysis highlights the trade-offs between model accuracy and runtime, emphasizing RoBERTa's suitability for scenarios prioritizing accuracy, and DistilBERT's potential in time-sensitive or resource-constrained applications. These findings support the integration of sentiment analysis into disaster response systems to enhance situational awareness and decision-making.

Downloads

Download data is not yet available.

References

U. Naseem, I. Razzak, M. Khushi, P. W. Eklund, and J. Kim, “COVIDSenti: A large-scale benchmark Twitter data set for COVID-19 sentiment analysis,” IEEE Transactions on Computational Social Systems, vol. 8, no. 4,pp. 1003–1015, Aug. 2021. doi : 10.1109/TCSS.2021.3051189.

G. Mu, J. Li, X. Li, C. Chen, X. Ju, and J. Dai, “An Enhanced IDBO-CNN-BiLSTM Model for Sentiment Analysis of Natural Disaster Tweets,” Biomimetics, vol. 9, no. 9, p. 533, 2024, doi: 10.3390/biomimetics9090533.

V. Lorini, E. Panizio, and C. Castillo, "SMDRM: A Platform to Analyze Social Media for Disaster Risk Management in Near Real Time," in Workshop Proceedings 16th International AAAI Conferences on Web and Social Media (ICWSM), 2022. doi: 10.36190/2022.1.

H. B. Firmansyah and A. Afriansyah, "Comparing Human and Machine-Labeled Sentiment Analysis for Disaster Response," in Proceedings 12th Multidisciplinary International Social Networks Conference (MISNC), 2025.

S. Z. Hassan, K. Ahmad, S. Hicks, P. Halvorsen, A. Al-Fuqaha, N. Conci, and M. Riegler, "Visual Sentiment Analysis from Disaster Images in Social Media," Sensors, vol. 22, no. 10, p. 3628, May 2022. doi: 10.3390/s22103628.

S. Behl, A. Rao, S. Aggarwal, S. Chadha, and H. S. Pannu, "Twitter for disaster relief through sentiment analysis for COVID-19 and natural hazard crises," International Journal of Disaster Risk Reduction., vol. 55, p. 102101, Mar. 2021. doi: 10.1016/j.ijdrr.2021.102101.

P. Y. W. Myint, S. L. Lo, and Y. Zhang, "Unveiling the dynamics of crisis events: Sentiment and emotion analysis via multi-task learning with attention mechanism and subject-based intent prediction," Information Processing & Management, vol. 61, no. 4, p. 103695, Jul. 2024. doi: 10.1016/j.ipm.2024.103695.

N. Kankanamge, T. Yigitcanlar, A. Goonetilleke, and M. Kamruzzaman, "Determining disaster severity through social media analysis: Testing the methodology with South East Queensland Flood tweets," International Journal of Disaster Risk Reduction, vol. 42, p. 101360, Jan. 2020. doi: 10.1016/j.ijdrr.2019.101360.

A. S. Hashim, N. Moorthy, A. A. Muazu, R. Wijaya, T. Purboyo, R. Latuconsina, C. Setianingsih, and M. F. Ruriawan, "Leveraging Social Media Sentiment Analysis for Enhanced Disaster Management: A Systematic Review and Future Research Agenda," Journal of System and Management Sciences, vol. 15, no. 4, pp. 170–191, 2025. doi: 10.33168/JSMS.2025.0412.

A. Areshey and H. Mathkour, "Exploring transformer models for sentiment classification: A comparison of BERT, RoBERTa, ALBERT, DistilBERT, and XLNet," Expert Systems, vol. 41, no. 11, e13701, 2024. doi: 10.1111/exsy.13701.

D. Contreras, S. Wilkinson, N. Balan, and P. James, "Assessing post-disaster recovery using sentiment analysis: The case of L’Aquila, Italy," Earthquake Spectra, vol. 38, no. 1, pp. 81–108, 2022. doi: 10.1177/87552930211036486.

M. Bouazizi and T. Ohtsuki, "Multi-class sentiment analysis on Twitter: Classification performance and challenges," Big Data Mining and Analytics, vol. 2, no. 3, pp. 181–194, Sep. 2019. doi: 10.26599/BDMA.2019.9020002.

A. R. Nair, R. P. Singh, D. Gupta, and P. Kumar, "Evaluating the Impact of Text Data Augmentation on Text Classification Tasks using DistilBERT," Procedia Computer Science, vol. 235, pp. 102–111, 2024. doi: 10.1016/j.procs.2024.05.102.

H. B. Firmansyah, J. L. Fernandez-Marquez, and J. Cerquides, “Ensemble learning for the classification of social media data in disaster response,” in Proceedings 19th International Conference on Information Systems for Crisis Response and Management (ISCRAM), Tarbes, France, May 2022, pp. 710–718.

H. B. Firmansyah, V. Lorini, M. O. Mulayim, J. Gomes, and J. L. Fernandez-Marquez, “Improving social media geolocation for disaster response by using text from images and ChatGPT,” in Proceedings 11th Multidisciplinary International Social Networks Conf. (MISNC), Bali, Indonesia, Aug. 2024, pp. 67–72. doi: 10.1145/3675669.3675696.

K. Seneviratne, M. Nadeeshani, S. Senaratne, and S. Perera, “Use of social media in disaster management: Challenges and strategies,” Sustainability, vol. 16, no. 11, pp.1-21, Art. no. 4824, 2024. doi: 10.3390/su16114824.

G. A. Ruz, P. A. Henríquez, and A. Mascareño, “Sentiment analysis of Twitter data during critical events through Bayesian networks classifiers,” Future Generation Computer Systems, vol. 106, pp.92-104, 2020, doi: 10.1016/j.future.2020.01.005.

G. Airlangga, “Comparative Analysis of Machine Learning Models for Real-Time Disaster Tweet Classification: Enhancing Emergency Response with Social Media Analytics,” Brilliance Research of Artificial Intelligence, vol. 4, no. 1, pp. 25–31, 2024, doi: 10.47709/brilliance.v4i1.3669.

M. A. Nulhakim, Y. Sibaroni, K. Muhammad, and N. Ku, “Geospatial Sentiment Analysis Using Twitter Data on Natural Disasters in Indonesia with Support Vector Machine,” International Journal on Information and Communication Technology,vol. 10, no. 2, pp. 242–258, 2024, doi: 10.21108/ijoict.v10i2.1032.

S. Vijayarani, M. J. Ilamathi, and M. Nithya, “Preprocessing Techniques for Text Mining-An Overview Privacy Preserving Data Mining View project,” Journal of Computer Science & Computer Networks, vol. 5, no. 1, pp. 7-16, 2015.

N. R. Paul, C. Rakesh, and D. Sahoo, "Fine-Tuning Transformer-Based Representations in Active Learning for Labelling Crisis Dataset of Tweets," SN Computer Science, vol.4, p. 553,2023, doi : doi.org/10.1007/s42979-023-02061-z

L. Peek, J. Tobin, R. M. Adams, H. Wu, and M. C. Mathews, "A Framework for Convergence Research in the Hazards and Disaster Field: The Natural Hazards Engineering Research Infrastructure Converge Facility," Frontiers in Built Environment, vol. 6, p. 110, 2020

R. Prabowo and M. Thelwall, “Sentiment analysis: A combined approach,” Journal Informetrics, vol. 3, no. 2, pp.143-157,2009, doi: 10.1016/j.joi.2009.01.003.

Pang, B., & Lee, L. “Opinion mining and sentiment analysis,” Foundations and Trends in Information Retrieval, vol. 2, no 1–2, pp. 1–135, 2008

M. Sari, S. Syahrullah, N. T. Lapatta, and R. Ardiansyah, "Twitter (X) Sentiment Analysis of Kampus Merdeka Program Using Support Vector Machine Algorithm and Selection Feature Chi-Square," Journal Teknik Informatika (JUTIF), vol. 5, no. 5, pp. 1249–1256, Oct. 2024, doi: 10.52436/1.jutif.2024.5.5.2037.

A. S. Talaat, “Sentiment analysis classification system using hybrid BERT models,” Journal of Big Data, vol. 10, no. 110, pp. 1–18, 2023. doi: 10.1186/s40537-023-00781-w.

R. Catelli, S. Pelosi, and M. Esposito, "Lexicon-Based vs. BERT-Based Sentiment Analysis: A Comparative Study in Italian," Electronics, vol. 11, no. 3, p. 374, Jan. 2022, doi: 10.3390/electronics11030374.

A. Joshy and S. Sundar, “Analyzing the Performance of Sentiment Analysis using BERT, DistilBERT, and RoBERTa,” in Proc. 2022 IEEE International Power and Renewable Energy Conference (IPRECON), 2022, pp. 1–6, doi: 10.1109/IPRECON55716.2022.10059542.

A. Irianti, H. Halimah, S. Sutedi, and M. Agariana, “Integration of BERT and SVM in Sentiment Analysis of Twitter/X Regarding Constitutional Court Decision No. 60/PUU‑XXII/2024,” Journal Teknik Informatika (JUTIF), vol. 6, no. 2, pp. 469–482, Apr. 2025, doi: 10.52436/1.jutif.2025.6.2.4068