TEXT CLASSIFICATION USING INDOBERT FINE-TUNING MODELING WITH CONVOLUTIONAL NEURAL NETWORK AND BI-LSTM

Alda Zevana; Dwiza Riana

doi:10.52436/1.jutif.2023.4.6.1650

Alda Zevana Computer Science, Department of Information Technology, Nusa Mandiri University, Indonesia
Dwiza Riana Computer Science, Department of Information Technology, Nusa Mandiri University, Indonesia

DOI: https://doi.org/10.52436/1.jutif.2023.4.6.1650

Keywords: Bi-LSTM, Convolutional Neural Network, Delivery, IndoBert Fine-Tuning, Twitter

Abstract

The technological advancements in goods delivery facilities have been increasing year by year in tandem with the growing online trade, which necessitates delivery services to fulfill the transactional process between sellers and buyers. Since 2000, top brand awards have often conducted official survey analyses to provide comparisons of goods or services, one of which includes delivery services. However, the survey rankings based on public opinion are less accurate due to users of delivery services and service companies being unaware of the specific success factors and weaknesses in their services. The aim of this research is to analyze the comparison of text mining using the Indonesian language transformation method, IndoBert. The algorithm utilized to demonstrate analysis performance employs Convolutional Neural Network (CNN) and Bidirectional Long Short-Term Memory (Bi-LSTM). This method is utilized to determine the impact of opinion data from Twitter on the J&T Express expedition delivery service, incorporating both text preprocessing and data without text preprocessing. The IndoBert parameters vary in the learning rate section based on four factors: price, time, returns, and others. The research data consisted of 2525 comments from Twitter users regarding the delivery service spanning from January 1, 2021, to March 31, 2023. The testing showed that Bi-LSTM with text preprocessing performed 2% higher, achieving 79% at a learning rate of 1x10^-6, compared to without text preprocessing at the same learning rate, which reached 77%. Additionally, CNN outperformed by 3% with a rate of 83%, compared to 80% without text preprocessing at a learning rate of 1x10^-5. The highest accuracy, reaching 83%, was obtained by CNN with parameters set at 1x10^-5, and the preprocessing technique was considered superior to Bi-LSTM.

Downloads

Download data is not yet available.

References

I. R. M. Association, Research Anthology on Social Media Advertising and Building Consumer Relationships. IGI Global, 2022.

S. Smith, Social Media Marketing for Business 2020. Samuel Smith, 2021.

N. F. Rahmah, “The Influence of Service Quality on Consumer satisfaction Delivery Services at JNT Express in Garut,” bircu, vol. 5, no. 3, p. 13, 2022, doi: https://doi.org/10.33258/birci.v5i3.6628.

F. Dwita, “THE EFFECT OF TOXIC LEADERSHIP AND JOB STRESS ON TURNOVER INTENTION IN LOGISTIC COURIER BEKASI CITY,” Airlangga J. Innov. Manag., vol. 3, no. 2, p. 10, 2022, doi: 10.20473/ajim.v3i1.39982.

A. George, Python Text Mining. BPB Publications, 2022.

J. Yang, “Brief introduction of medical database and data mining technology in big data era,” J. Evid. Based. Med., vol. 13, no. 1, pp. 57–69, 2020, doi: https://doi.org/10.1111/jebm.12373.

K. You, “HOW DOES LEARNING RATE DECAY HELP MODERN NEURAL NETWORKS?,” Mach. Learn., vol. 2, p. 15, 2019, doi: https://doi.org/10.48550/arXiv.1908.01878.

F. He, “Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence,” p. 4, 2019.

K. K. Chandriah, “RNN / LSTM with modified Adam optimizer in deep learning approach for automobile spare parts demand forecasting,” SpringerLink, 2021.

A. Glassner, Deep Learning : A Visual Approach. Amerika Serikat: No Starch Press, 2021.

I. Kandel, “The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset,” ScienceDirect, vol. 6, p. 4, 2020, doi: https://doi.org/10.1016/j.icte.2020.04.010.

N. Aneja, “Transfer Learning using CNN for Handwritten Devanagari Character Recognition,” IEEE, 2019, doi: https://doi.org/10.1109/ICAIT47043.2019.8987286.

L. F. W. Anthony, “Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models,” Comput. Soc., vol. 3, p. 11, 2020, doi: https://doi.org/10.48550/arXiv.2007.03051.

A. Labach, “Survey of Dropout Methods for Deep Neural Networks,” Neural Evol. Comput., vol. 2, p. 13, 2019, doi: https://doi.org/10.48550/arXiv.1904.13310.

B. A. Skourt, “Mixed-pooling-dropout for convolutional neural network regularization,” J. King Saud Univ. – Comput. Inf. Sci., vol. 34, no. 8, p. 6, 2022, doi: https://doi.org/10.1016/j.jksuci.2021.05.001.

S. Alaparthi, “Bidirectional Encoder Representations from Transformers (BERT): A sentiment analysis odyssey,” Comput. Lang., p. 15, 2020, doi: https://doi.org/10.48550/arXiv.2007.01127.

A. S. A. Ali, “CNN-Bi LSTM Neural Network for Simulating Groundwater Level,” Comput. Res. Prog. Appl. Sci. Eng., p. 7, 2022, doi: https://doi.org/10.52547/crpase.8.1.2748.

M. Pota, “An Effective BERT-Based Pipeline for Twitter Sentiment Analysis: A Case Study in Italian,” Sensors, p. 21, 2020, doi: https://doi.org/10.3390/s21010133.

L. P. M. Aliyah Kurniasih, “On the Role of Text Preprocessing in BERT Embedding-based DNNs for Classifying Informal Texts,” Aliyah Kurniasih, vol. 13, p. 8, 2022.

W. Lu, “A CNN-BiLSTM-AM method for stock price prediction,” Neural Comput. Appl., vol. 33, no. 4, pp. 4741–4753, 2020, doi: https://doi.org/10.1007/s00521-020-05532-z.

F. Theis, Artificial Neural Networks and Machine Learning – ICANN 2019: Text and Time Series. Springer International Publishing, 2019.

M. Tan, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” Int. Conf. Mach. Learn. Proc. Mach. Learn. Res., vol. 97, pp. 6105–6114, 2019.

S. Z. Pranida, “Sentiment Analysis of Expedition Customer Satisfaction using BiGRU and BiLSTM,” IJAIDM, vol. 5, no. 1, p. 10, 2022.