SENTIMENT ANALYSIS OF CHATGPT APP USER REVIEWS USING SVM AND CNN METHODS

Nurul  Widaad; Dina Anggraini

doi:10.52436/1.jutif.2024.5.6.4010

Nurul Widaad Departement of Information System, Technology and Engineering Faculty, Universitas Gunadarma, Indonesia
Dina Anggraini Departement of Information System, Technology and Engineering Faculty, Universitas Gunadarma, Indonesia

DOI: https://doi.org/10.52436/1.jutif.2024.5.6.4010

Keywords: Artificial Intelligence, Convolutional Neural Network, Sentiment Analysis, Support Vector Machine

Abstract

The rapid development of Artificial Intelligence (AI) has significantly impacted various sectors, including user interactions with natural language-based applications such as ChatGPT. This study aims to analyze user sentiment towards ChatGPT amidst the emergence of alternative AI technologies like Gemini (Google Deep Mind), Claude (Anthropic AI), and LLaMA (Meta AI). ChatGPT was chosen as the research subject due to its role as a pioneer in public AI usage. The research focuses on uncovering user sentiments—positive, negative, or neutral. A total of 155,529 reviews from the Google Play Store were analyzed using Support Vector Machine (SVM) and Convolutional Neural Network (CNN) algorithms. The research process involved data collection (data scraping), preprocessing (emoji removal, case folding, punctuation removal, tokenization, stopword removal, stemming, and normalization), sentiment labeling, data splitting (80% training and 20% testing), and model evaluation using accuracy, precision, recall, and F1-score metrics. The results indicate that the SVM model achieved an accuracy of 85%, with a precision of 0.83, recall of 0.55, and an F1-score of 0.58. Meanwhile, the CNN model recorded an accuracy of 84%, with a precision of 0.68, recall of 0.59, and an F1-score of 0.62. Among the analyzed reviews, 75% expressed positive sentiment, 18.22% negative, and 6.71% neutral. The dominance of positive sentiment reaffirms ChatGPT's position as a preferred choice among users, although certain aspects need improvement to maintain its competitiveness amidst growing AI competition. This study provides valuable insights for developers to identify the strengths and weaknesses of ChatGPT based on user feedback, enabling them to optimize the application's features to create a more satisfying and relevant user experience in the future.

Downloads

Download data is not yet available.

References

P. Kecerdasan Buatan Dalam Mendukung Pembelajaran dan B. Karyadi, “PEMANFAATAN KECERDASAN BUATAN DALAM MENDUKUNG PEMBELAJARAN MANDIRI,” JurnalTeknologiPendidikan, vol. 8, no. 2, hlm. 253–258, Jul 2023, doi: 10.32832/educate.v8i02.14843.

J. Carvalko, J. R. Carvalko, dan J. Ieee, “GPT-A Paradigm Shift for the Twenty-First Century,” Okt 2023, doi: 10.36227/techrxiv.23690874.v1.

B. Gordijn dan H. ten Have, “ChatGPT: evolution or revolution?,” 1 Maret 2023, Springer Science and Business Media B.V. doi: 10.1007/s11019-023-10136-0.

P. Chakriswaran, D. R. Vincent, K. Srinivasan, V. Sharma, C. Y. Chang, dan D. G. Reina, “Emotion AI-driven sentiment analysis: A survey, future research directions, and open issues,” 1 Desember 2019, MDPI AG. doi: 10.3390/app9245462.

M. Özel dan Ö. Çetinkaya Bozkurt, “Sentiment Analysis on GPT-4 with Comparative Models Using Twitter Data,” Acta Infologica, vol. 8, no. 1, hlm. 23–33, Jun 2024, doi: 10.26650/acin.1418834.

L. Alzubaidi dkk., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” J Big Data, vol. 8, no. 53, Des 2021, doi: 10.1186/s40537-021-00444-8.

I. M. Karo Karo, J. A. Karo Karo, Y. Yunianto, H. Hariyanto, M. Falah, dan M. Ginting, “Analisis Sentimen Ulasan Aplikasi Info BMKG di Google Play Menggunakan TF-IDF dan Support Vector Machine,” Journal of Information System Research (JOSH), vol. 4, no. 4, hlm. 1423–1430, Jul 2023, doi: 10.47065/josh.v4i4.3943.

M. K. Dahouda dan I. Joe, “A Deep-Learned Embedding Technique for Categorical Features Encoding,” IEEE Access, vol. 9, hlm. 114381–114391, 2021, doi: 10.1109/ACCESS.2021.3104357.

M. F. Fakhrezi, Adian Fatchur Rochim, dan Dinar Mutiara Kusomo Nugraheni, “Comparison of Sentiment Analysis Methods Based on Accuracy Value Case Study: Twitter Mentions of Academic Article,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 7, no. 1, hlm. 161–167, Feb 2023, doi: 10.29207/resti.v7i1.4767.

G. M. Foody, “Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient,” PLoS One, vol. 18, no. 10, hlm. 1–27, Okt 2023, doi: 10.1371/journal.pone.0291908.

D. Krstinić, M. Braović, L. Šerić, dan D. Božić-Štulić, “Multi-label Classifier Performance Evaluation with Confusion Matrix,” Academy and Industry Research Collaboration Center (AIRCC), Jun 2020, hlm. 1–14. doi: 10.5121/csit.2020.100801.

Naruaki Ogasawara, “Visualization of Research Topics Using Word Cloud Technology, Characteristics of Research Trends in Article Titles and Keywords,” The Japanese Society of Internal Medicine (JSIM), Mei 2024, doi: 10.51094/jxiv.722.

F. Setya Ananto dan F. N. Hasan, “Implementasi Algoritma Naïve Bayes Terhadap Analisis Sentimen Ulasan Aplikasi MyPertamina pada Google Play Store,” Jurnal ICT : Information Communication & Technology, vol. 23, no. 1, hlm. 75–80, Jul 2023, [Daring]. Tersedia pada: https://ejournal.ikmi.ac.id/index.php/jict-ikmi

K. Maharana, S. Mondal, dan B. Nemade, “A review: Data pre-processing and data augmentation techniques,” Global Transitions Proceedings, vol. 3, no. 1, hlm. 91–99, Jun 2022, doi: 10.1016/j.gltp.2022.04.020.

M. A. Rosid, A. S. Fitrani, I. R. I. Astutik, N. I. Mulloh, dan H. A. Gozali, “Improving Text Preprocessing for Student Complaint Document Classification Using Sastrawi,” dalam IOP Conference Series: Materials Science and Engineering, Institute of Physics Publishing, Jul 2020. doi: 10.1088/1757-899X/874/1/012017.

Rianto, A. B. Mutiara, E. P. Wibowo, dan P. I. Santosa, “Improving the accuracy of text classification using stemming method, a case of non-formal Indonesian conversation,” J Big Data, vol. 8, no. 26, hlm. 1–16, Des 2021, doi: 10.1186/s40537-021-00413-1.