COMPARISON OF SUPPORT VECTOR MACHINE AND INDOBERT IN NON-FUNCTIONAL REQUIREMENT CLASSIFICATION OF APPLICATION USER REVIEWS
Abstract
User reviews of mobile applications have become a valuable source of information for evaluating the quality of an application. It is crucial for application developers to understand what users express in their reviews. One aspect that can be analyzed from user reviews is Non-Functional Requirement (NFR). Classifying reviews based on NFR is essential in understanding how an application can be enhanced. Although user reviews have the potential to provide valuable insights into NFR, manually processing thousands of user reviews is a laborious and inefficient task. Therefore, artificial intelligence methods are employed to automatically classify user reviews into relevant NFR categories. This research discusses the performance comparison of the SVM and IndoBERT algorithms in NFR classification. The study involves collecting application review data from 2018 to 2023, sourced from Google Playstore and Apple Appstore, followed by annotating the review data based on ISO 25010. Subsequently, the data is allocated into training and testing sets with an 80:20 ratio. Further, a data preprocessing phase is conducted, which includes steps such as lowercasing, tokenization, special character removal, text normalization, and text stemming. The next step involves training the SVM and IndoBERT algorithms on the dataset. Finally, the evaluation is carried out by calculating the F1-score. The research results indicate that the IndoBERT model outperforms the SVM model. The IndoBERT algorithm excels in recognizing NFR in reviews, achieving an F1-score of 93%, while the SVM algorithm achieves an F1-score of 91%.
Downloads
References
Badan Pusat Statistik, Statistik Telekomunikasi Indonesia 2022. Jakarta: Badan Pusat Statistik, 2023.
M. Nixon, M. Jose, and A. K. Asst Professor, “App store bugs-review classification using BERT- DNN model,” Turkish J. Comput. Math. Educ., vol. 12, no. 13, pp. 5300–5306, 2021, [Online]. Available: https://www.turcomat.org/index.php/turkbilmat/article/view/9721.
D. Dave and V. Anu, “Identifying Functional and Non-functional Software Requirements From User App Reviews,” in 2022 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS), 2022, pp. 1–6, doi: 10.1109/IEMTRONICS55184.2022.9795770.
J. Y. Lee, “User Review Mining: An Approach for Software Requirements Evolution,” Int. J. Adv. Smart Converg., vol. 9, no. 4, pp. 124–131, 2020, [Online]. Available:http://dx.doi.org/10.7236/IJASC.2020.9.4.124.
Y. Yao, W. Jiang, Y. Wang, P. Song, and B. Wang, “Non-Functional Requirements Analysis Based on Application Reviews in the Android App Market,” Inf. Resour. Manag. J., vol. 35, no. 2, pp. 1–17, 2022, [Online]. Available: https://ideas.repec.org/a/igg/rmj000/v35y2022i2p1-17.html.
M. Zarlis, Elviwani, A. Dilham, Relita, and V. Yasin, Kecerdasan Buatan Era Revolusi Industri 4.0, 1st ed. Bogor: Mitra Wacana Media, 2022.
Preeti, “Review on Text Mining: Techniques, Applications and Issues,” Proc. 2021 10th Int. Conf. Syst. Model. Adv. Res. Trends, SMART 2021, vol. 7, no. 11, pp. 474–478, 2021, doi: 10.1109/SMART52563.2021.9676285.
Deng, L., Liu, Y. Deep Learning in Natural Language Processing, Springer, Singapore. 2022. https://doi.org/10.1007/978-981-10-5209-5_11
R. dos Santos, K. Villela, D. T. Avila, and L. H. Thom, “A practical user feedback classifier for software quality characteristics,” Proc. Int. Conf. Softw. Eng. Knowl. Eng. SEKE, vol. 2021-July, no. May, pp. 340–345, 2021, doi: 10.18293/SEKE2021-055.
N. Al Kilani, R. Tailakh, and A. Hanani, “Automatic Classification of Apps Reviews for Requirement Engineering: Exploring the Customers Need from Healthcare Applications,” in 2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS), 2019, pp. 541–548, doi: 10.1109/SNAMS.2019.8931820.
M. Haoues, R. Mokni, and A. Sellami, “Machine learning for mHealth apps quality evaluation,” Software Quality Journal, May 2023, doi: https://doi.org/10.1007/s11219-023-09630-8.
B. Rehman, K.A. Alam, K.M., Ko." Automated Classification of Mobile App Reviews Considering User’s Quality Concerns," Advances in Intelligent Systems and Computing, vol. 1357, June 2021, doi: https://doi.org/10.1007/978-981-16-1045-5_3
S. A. Amira and M. I. Irawan, “Opinion Analysis of Traveler Based on Tourism Site Review Using Sentiment Analysis,” IPTEK The Journal for Technology and Science, vol. 31, no. 2, p. 223, May 2020, doi: https://doi.org/10.12962/j20882033.v31i2.6338.
J. Cardoso-Fernandes, A. C. Teodoro, A. Lima, and E. Roda-Robles, “Semi-Automatization of Support Vector Machines to Map Lithium (Li) Bearing Pegmatites,” Remote Sensing, vol. 12, no. 14, p. 2319, Jul. 2020, doi: https://doi.org/10.3390/rs12142319.
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” NAACL HLT 2019 - 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. - Proc. Conf., vol. 1, pp. 4171–4186, 2019.
B. Wilie et al., “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” 2020, [Online]. Available: http://arxiv.org/abs/2009.05387.
K. Pohl and C. Rupp, Requirements engineering fundamentals : a study guide for the certified professional for requirements engineering exam ; foundation level - IREB compliant. Santa Barbara, Calif.: Rocky Nook, 2015.
A. Kulkarni, NATURAL LANGUAGE PROCESSING RECIPES : unlocking text data with machine learning and deep ... learning using python. 2019.
S. Mokhtari, K. K. Yen, and J. Liu, “Effectiveness of Artificial Intelligence in Stock Market Prediction based on Machine Learning,” International Journal of Computer Applications, vol. 183, no. 7, pp. 1–8, Jun. 2021, doi: https://doi.org/10.5120/ijca2021921347
Copyright (c) 2024 Abdul Ghofur Rais Kumar, Yudi Sukmono, Aji Ery Burhandenny
This work is licensed under a Creative Commons Attribution 4.0 International License.
