PERFORMANCE OF TEXT SIMILARITY ALGORITHMS FOR ESSAY ANSWER SCORING IN ONLINE EXAMINATIONS
Abstract
The purpose of assessment is to determine learning success. Exams with question descriptions have several advantages, including ease of preparation and the ability to reveal student comprehension and originality. The problem with space is that it takes time to fix. Therefore, it is important to develop algorithms and software that automatically evaluate space. With the help of this algorithm and this software, you can solve some exam and assessment problems. This study aims to investigate similarity algorithms that approximate human patterns in evaluating ambiguous answers. This study examines his five similarity algorithms, including TF-IDF and LSA. The data was a collection of correct answers with a total of 371 texts. The similarity algorithm's performance was compared with human correction results. Evaluation was performed using Root Mean Square Error (RMSE). This study shows that his TF-IDF algorithm like Jaccard has the lowest his RMSE compared to human judgement. However, the LSA algorithm tended better to follow human rating patterns for descriptive tests..
Downloads
References
D. Yulianto and A. S. Nugraheni, “Efektivitas Pembelajaran Daring Dalam Pembelajaran Bahasa Indonesia,” Decod. J. Pendidik. Teknol. Inf., vol. 1, no. 1, pp. 33–42, Mar. 2021, doi: 10.51454/decode.v1i1.5.
P. Paudel, “Online education: Benefits, challenges and strategies during and after COVID-19 in higher education,” Int. J. Stud. Educ., vol. 3, no. 2, pp. 70–85, 2021.
N. Sabani, “Pembelajaran Daring Menghadapi Fenomena Pandemi Covid-19 (Systematic Literature Review),” J. Psychol. Treat., vol. 1, no. 1, pp. 11–21, 2021.
I. Faza Ahmad, “ALTERNATIVE ASSESSMENT IN DISTANCE LEARNING IN EMERGENCIES SPREAD OF CORONAVIRUS DISEASE (COVID-19) IN INDONESIA,” J. Pedagog., vol. 07, no. 01, 2020, [Online]. Available: https://ejournal.unuja.ac.id/index.php/pedagogik
R. D. Mahande, F. A. Darmawan, and ..., “Customization and Usability Testing Auto Essay File Grading Lms Based Metacognitive Assessment in Engineering Faculty,” … Tek. Inform., vol. 3, no. 2, 2022, [Online]. Available: http://jutif.if.unsoed.ac.id/index.php/jurnal/article/view/118%0Ahttp://jutif.if.unsoed.ac.id/index.php/jurnal/article/download/118/65
S. Vajjala, “Automated Assessment of Non-Native Learner Essays: Investigating the Role of Linguistic Features,” Int. J. Artif. Intell. Educ., vol. 28, no. 1, pp. 79–105, Mar. 2018, doi: 10.1007/s40593-017-0142-3.
M. S. Fahlevi and L. S. Zanthy, “Analisis Kesulitan Siswa dalam Menyelesaikan Soal Uraian pada Materi Bangun Ruang Sisi Datar,” J. Pembelajaran Mat. Inov., vol. 3, no. 4, 2020, doi: 10.22460/jpmi.v3i4.313-322.
H. Suwandi, H. Harlinda, and S. H. Mansyur, “Implementation of a School Information System Using Rapid Application Development Method,” J. Tek. Inform., vol. 3, no. 6, pp. 1501–1512, 2022, doi: 10.20884/1.jutif.2022.3.6.332.
J. Zeniarta, A. Salam, and I. Achsamu, “Sistem Koreksi Jawaban Esai Otomatis (E-Valuation) dengan Vector Space Model pada Computer Based Test (CBT),” Semin. Nas. Din. Inform., pp. 91–96, 2020.
A. Rokade, B. Patil, S. Rajani, S. Revandkar, and R. Shedge, “Automated Grading System Using Natural Language Processing,” in Proceedings of the International Conference on Inventive Communication and Computational Technologies, ICICCT 2018, Sep. 2018, pp. 1123–1127. doi: 10.1109/ICICCT.2018.8473170.
A. A. Putri Ratna, A. Kaltsum, L. Santiar, H. Khairunissa, I. Ibrahim, and P. D. Purnamasari, “Term Frequency-Inverse Document Frequency Answer Categorization with Support Vector Machine on Automatic Short Essay Grading System with Latent Semantic Analysis for Japanese Language,” in ICECOS 2019 - 3rd International Conference on Electrical Engineering and Computer Science, Proceeding, 2019, pp. 293–298. doi: 10.1109/ICECOS47637.2019.8984530.
H. Thamrin, N. A. Verdikha, and A. Triyono, “Text Classification and Similarity Algorithms in Essay Grading,” in 2021 4th International Seminar on Research of Information Technology and Intelligent Systems, ISRITI 2021, 2021, pp. 201–205. doi: 10.1109/ISRITI54043.2021.9702808.
M. Jamaluddin, N. Yuniarti, A. Rahmani, and J. Hutahaean, “Aplikasi Penilaian Otomatis Ujian Esai Berbahasa Indonesia Menggunakan Algoritma K-Nearest Neighbor ( Studi kasus MAN Cimahi ),” Pros. Ind. Res. Work. Natl. Semin., vol. 10, no. August 2019, pp. 314–324, 2020, [Online]. Available: https://jurnal.polban.ac.id/ojs-3.1.2/proceeding/article/view/1404
V. Nandini and P. Uma Maheswari, “Automatic assessment of descriptive answers in online examination system using semantic relational features,” J. Supercomput., vol. 76, no. 6, pp. 4430–4448, Jun. 2020, doi: 10.1007/s11227-018-2381-y.
S. M. C. Loureiro, J. Guerreiro, S. Eloy, D. Langaro, and P. Panchapakesan, “Understanding the use of Virtual Reality in Marketing: A text mining-based review,” J. Bus. Res., vol. 100, pp. 514–530, Jul. 2019, doi: 10.1016/j.jbusres.2018.10.055.
F. Sutomo et al., “OPTIMIZATION OF THE K-NEAREST NEIGHBORS ALGORITHM USING THE OPTIMASI ALGORITMA K-NEAREST NEIGHBORS MENGGUNAKAN METODE,” vol. 4, no. 1, pp. 125–130, 2023.
M. Milkhatun, A. A. F. Rizal, N. W. W. Asthiningsih, and A. J. Latipah, “Performance Assessment of University Lecturers: A Data Mining Approach,” Khazanah Inform. J. Ilmu Komput. dan Inform., vol. 6, no. 2, 2020.
R. Rosnelly, D. Hartama, M. Sadikin, C. P. Lubis, M. S. Simanjuntak, and S. Kosasi, “The Similarity of Essay Examination Results using Preprocessing Text Mining with Cosine Similarity and Nazief-Adriani Algorithms,” Turkish J. Comput. Math. Educ., vol. 12, no. 3, pp. 1415–1422, Apr. 2021, doi: 10.17762/TURCOMAT.V12I3.938.
R. Riyanto, “Implementation of the Jaccard Similarity Algorithm on Answer Type Description,” IJIIS Int. J. Informatics Inf. Syst., vol. 5, no. 2, pp. 76–83, 2022, doi: 10.47738/ijiis.v5i2.130.
H. Zhou, “Research of Text Classification Based on TF-IDF and CNN-LSTM,” J. Phys. Conf. Ser., vol. 2171, no. 1, 2022, doi: 10.1088/1742-6596/2171/1/012021.
N. Hidayat and L. Afuan, “Penilaian Ujian Otomatis untuk Soal Bertipe Essay pada PJJ APTIKOM menggunakan Cosine Similarity,” Semin. Nas. APTIKOM, pp. 259–271, 2019.
H. Fan and Y. Qin, “Research on Text Classification Based on Improved TF-IDF Algorithm,” vol. 147, no. Ncce, pp. 501–506, 2018, doi: 10.2991/ncce-18.2018.79.
V. V. Ramalingam, A. Pandian, P. Chetry, and H. Nigam, “Automated Essay Grading using Machine Learning Algorithm,” J. Phys. Conf. Ser., vol. 1000, no. 1, 2018, doi: 10.1088/1742-6596/1000/1/012030.
G. I. Marthasari, N. Hayatin, and M. Yuniarti, “Content Classification based-on Latent Semantic Analysis and Support Vector Machine (LSA-SVM),” J. Transform., vol. 19, no. 2, p. 144, 2022, doi: 10.26623/transformatika.v19i2.2745.
J. Hoblos, “Experimenting with latent semantic analysis and latent dirichlet allocation on automated essay grading,” in 2020 Seventh International Conference on Social Networks Analysis, Management and Security (SNAMS), 2020, pp. 1–7.
A. A. Putri Ratna, H. Khairunissa, A. Kaltsum, I. Ibrahim, and P. D. Purnamasari, “Automatic Essay Grading for Bahasa Indonesia with Support Vector Machine and Latent Semantic Analysis,” in ICECOS 2019 - 3rd International Conference on Electrical Engineering and Computer Science, Proceeding, 2019, pp. 363–367. doi: 10.1109/ICECOS47637.2019.8984528.
A. A. P. Ratna, N. A. Wulandari, A. Kaltsum, I. Ibrahim, and P. D. Purnamasari, “Answer categorization method using K-means for Indonesian language automatic short answer grading system based on latent semantic analysis,” in 2019 16th International Conference on Quality in Research (QIR): International Symposium on Electrical and Computer Engineering, 2019, pp. 1–5.
Copyright (c) 2023 Muhammad Riza Radyaka Susanto, Husni Thamrin, Naufal Azmi Verdikha
This work is licensed under a Creative Commons Attribution 4.0 International License.
