Validation of Question Classification Using Support Vector Machine and Intraclass Correlation Coefficient Based on the Revised Bloom’s Taxonomy

Lazuardy Syahrul  Darfiansa; Sza Sza Amulya  Larasti

doi:10.52436/1.jutif.2025.6.6.4728

Authors

Lazuardy Syahrul Darfiansa Informatics Telkom University, Indonesia
Sza Sza Amulya Larasti Computer Science Brawijaya University, Indonesia

DOI:

https://doi.org/10.52436/1.jutif.2025.6.6.4728

Keywords:

ICC, Machine Learning, Question Classification, Revised Bloom Taxonomy, SVM

Abstract

The assessment process must be carried out accurately as it is a crucial aspect of identifying cognitive abilities in students. Cognitive ability identification needs to be done by providing exam questions that refer to the Revised Bloom's Taxonomy for difficulty-level classification to ensure students' understanding of what has been taught. The traditional manual classification process carried out by educators often requires significant time and is susceptible to subjective variability. The classification of questions from levels C1 to C6 based on the Revised Bloom's Taxonomy shows an imbalance in the data distribution for each level, leading to inaccurate classification results. The automatic classification technique using the SVM algorithm allows educators to quickly classify questions based on their difficulty levels. The automated classification technique needs to be validated to what extent the difficulty levels classified by the machine align with the perceptions of educators and students. This research will validate the results of question classification generated from the SVM algorithm, supplemented by the oversampling technique to address data imbalance. The validation method used is ICC. Applying the SMOTE oversampling technique to handle a class imbalance in the training data shows improvement, with an accuracy rate of 91% when using SMOTE compared to 83% without it. Results of the classification suitability test with the SVM algorithm by educators and students indicate a high level of agreement. The ICC Average Measures values are as follows: SVM classification is 0,979, assessment by non-science subject educators is 0,956, assessment by science subject educators is 0,991, assessment by non-science subject students is 0,982, and assessment by science subject students is 0,984. ICC testing consistently yields excellent results in non-science and science subjects, indicating that the assessments conducted by educators and students have a very high level of agreement.

Downloads

Download data is not yet available.

References

E. Ita, “Manajemen Pembelajaran Pendidikan Anak Usia Dini di TK Rutosoro Kecamatan Golewa Kabupaten Ngada Flores Nusa Tenggara Timur,” J. Dimensi Pendidik. Dan Pembelajaran, vol. 6, Jun. 2018, doi: 10.24269/dpp.v6i1.889.

E. R. Setyaningsih and I. Listiowarni, “Categorization of Exam Questions based on Bloom Taxonomy using Naïve Bayes and Laplace Smoothing,” in 2021 3rd East Indonesia Conference on Computer and Information Technology (EIConCIT), 2021, pp. 330–333. doi: 10.1109/EIConCIT50028.2021.9431862.

A. Sangodiah, T. J. San, Y. T. Fui, L. E. Heng, R. K. Ayyasamy, and N. B. A. Jalil, “Identifying Optimal Baseline Variant of Unsupervised Term Weighting in Question Classification Based on Bloom Taxonomy,” Mendel, vol. 28, no. 1, pp. 8–22, 2022, doi: 10.13164/mendel.2022.1.008.

I. Magdalena, N. F. Islami, E. A. Rasid, and N. T. Diasty, “Tiga Ranah Taksonomi Bloom dalam Pendidikan,” J. Edukasi Dan Sains, vol. 2, no. 1, pp. 132-`139, Jun. 2020, doi: 10.36088/EDISI.V2I1.822.

E. Subiyantoro, A. Ashari, and Suprapto, “Cognitive Classification Based on Revised Bloom’s Taxonomy Using Learning Vector Quantization,” in CENIM 2020 - Proceeding: International Conference on Computer Engineering, Network, and Intelligent Multimedia 2020, Institute of Electrical and Electronics Engineers Inc., Nov. 2020, pp. 349–353. doi: 10.1109/CENIM51130.2020.9297879.

I. I. Sinan, V. Nwoacha, J. Degila, and S. A. Onashoga, “A Comparison of Data-Driven and Data-Centric Architectures using E-Learning Solutions,” Proceedings - International Conference Advancement in Data Science, E-Learning and Information Systems, ICADEIS 2022, 2022, doi: 10.1109/ICADEIS56544.2022.10037358.

K. Makhlouf, L. Amouri, N. Chaabane, and N. El-Haggar, “Exam Questions Classification Based on Bloom’s Taxonomy: Approaches and Techniques,” in 2020 2nd International Conference on Computer and Information Sciences, ICCIS 2020, Institute of Electrical and Electronics Engineers Inc., Oct. 2020. doi: 10.1109/ICCIS49240.2020.9257698.

H. Shi et al., “Educational management in Critical Thinking Training Based on Bloom’s Taxonomy and SOLO Taxonomy,” Proc. - 2020 Int. Conf. Inf. Sci. Educ. ICISE-IE 2020, pp. 518–521, Dec. 2020, doi: 10.1109/ICISE51755.2020.00116.

H. Sharma, R. Mathur, T. Chintala, S. Dhanalakshmi, and R. Senthil, “An effective deep learning pipeline for improved question classification into bloom’s taxonomy’s domains,” Educ Inf Technol (Dordr), vol. 28, no. 5, pp. 5105–5145, May 2023, doi: 10.1007/s10639-022-11356-2.

M. O. Gani, R. K. Ayyasamy, A. Sangodiah, and Y. T. Fui, “Bloom’s Taxonomy-based exam question classification: The outcome of CNN and optimal pre-trained word embedding technique,” Educ Inf Technol (Dordr), vol. 28, no. 12, pp. 15893–15914, Dec. 2023, doi: 10.1007/s10639-023-11842-1.

L. S. Darfiansa and F. A. Bachtiar, “Comparative Analysis of Term Weighting Methods for Question Classification in Bloom Taxonomy Using Machine Learning Approach,” in 2023 IEEE International Conference on Computing (ICOCO), IEEE, Oct. 2023, pp. 259–264. doi: 10.1109/ICOCO59262.2023.10397821.

S. Basuki, Z. Rakhmawati, and G. W. Wicaksono, “Klasifikasi Kalimat Tanya Berdasarkan Taksonomi Bloom Menggunakan Support Vector Machine,” J. Repos., vol. 2, no. 4, pp. 427–436, 2020, doi: https://doi.org/10.22219/repositor.v2i4.30516.

A. Mohasseb, M. Bader-El-Den, M. Cocea, and H. A. N. Liu, “Improving Imbalanced Question Classification Using Structured Smote Based Approach,” in 2018 International Conference on Machine Learning and Cybernetics (ICMLC), 2018, pp. 593–597. doi: 10.1109/ICMLC.2018.8527028.

M. Ifham, K. Banujan, B. T. G. S. Kumara, and P. M. A. K. Wijeratne, “Automatic Classification of Questions based on Bloom’s Taxonomy using Artificial Neural Network,” 2022 International Conference on Decision Aid Sciences and Applications, DASA 2022, pp. 311–315, 2022, doi: 10.1109/DASA54658.2022.9765190.

N. Barari, M. RezaeiZadeh, A. Khorasani, and F. Alami, “Designing and validating educational standards for E-teaching in virtual learning environments (VLEs), based on revised Bloom’s taxonomy,” Interactive Learning Environments, vol. 30, no. 9, pp. 1640–1652, 2022, doi: 10.1080/10494820.2020.1739078.

A. A. Yahya, Z. Toukal, and A. Osman, “Bloom’s Taxonomy–Based Classification for Item Bank Questions Using Support Vector Machines,” Stud. Comput. Intell., vol. 431, pp. 135–140, 2012, doi: 10.1007/978-3-642-30732-4_17.

K. Jayakodi, M. Bandara, and I. Perera, “An automatic classifier for exam questions in Engineering: A process for Bloom’s taxonomy,” in 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), 2015, pp. 195–202. doi: 10.1109/TALE.2015.7386043.

L. S. Darfiansa, F. Azzuri, F. A. Bachtiar, and D. E. Ratnawati, “Comparative Analysis of Deep Learning and Machine Learning Techniques for Question Classification in Bloom’s Taxonomy,” 2023 1st International Conference on Advanced Engineering and Technologies, ICONNIC 2023 - Proceeding, pp. 103–108, 2023, doi: 10.1109/ICONNIC59854.2023.10467502.

M. Amien, “Sejarah dan Perkembangan Teknik Natural Language Processing (NLP) Bahasa Indonesia: Tinjauan tentang sejarah, perkembangan teknologi, dan aplikasi NLP dalam bahasa Indonesia,” ELANG J. Interdiscip. Res., vol. 1, no. 01, pp. 99–105, Aug. 2023, doi: 10.32664/ELANG.V1I01.

Hasmawati, A. Romadhony, and R. Abdurohman, “Primary and High School Question Classification based on Bloom’s Taxonomy,” in 2022 10th International Conference on Information and Communication Technology, ICoICT 2022, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 234–239. doi: 10.1109/ICoICT55009.2022.9914842.

A. H. Ababneh, “Investigating the relevance of Arabic text classification datasets based on supervised learning,” Journal of Electronic Science and Technology, vol. 20, no. 2, p. 100160, Jun. 2022, doi: 10.1016/J.JNLEST.2022.100160.

X. Luo, “Efficient English text classification using selected Machine Learning Techniques,” Alexandria Engineering Journal, vol. 60, no. 3, pp. 3401–3409, Jun. 2021, doi: 10.1016/J.AEJ.2021.02.009.

T. Sabri, O. El Beggar, and M. Kissi, “Comparative study of Arabic text classification using feature vectorization methods,” Procedia Comput Sci, vol. 198, pp. 269–275, 2022, doi: https://doi.org/10.1016/j.procs.2021.12.239.

M. Mohammed and N. Omar, “Question Classification Based on Bloom’s Taxonomy Using Enhanced TF-IDF,” Int. J. Adv. Sci. Eng. Inf. Technol., vol. 8, no. 4–2, pp. 1679–1685, Sep. 2018, doi: 10.18517/IJASEIT.8.4-2.6835.

D. Campos, R. R. Silva, and J. Bernardino, “Text mining in hotel reviews: Impact of words restriction in text classification,” in IC3K 2019 - Proceedings of the 11th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, SciTePress, 2019, pp. 442–449. doi: 10.5220/0008346904420449.

A. S. Callista, O. N. Pratiwi, and E. Sutoyo, “Questions Classification Based on Revised Bloom’s Taxonomy Cognitive Level using Naive Bayes and Support Vector Machine,” in Proceedings - 2021 4th International Conference on Computer and Informatics Engineering: IT-Based Digital Industrial Innovation for the Welfare of Society, IC2IE 2021, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 260–265. doi: 10.1109/IC2IE53219.2021.9649187.

M. Mohammedid and N. Omar, “Question classification based on Bloom’s taxonomy cognitive domain using modified TF-IDF and word2vec,” PLoS One, vol. 15, no. 3, 2020, doi: 10.1371/journal.pone.0230442.

D. H. Ismunarti, M. Zainuri, D. N. Sugianto, and S. W. Saputra, “Pengujian Reliabilitas Instrumen Terhadap Variabel Kontinu Untuk Pengukuran Konsentrasi Klorofil- A Perairan,” Buletin Oseanografi Marina, vol. 9, no. 1, pp. 1–8, Apr. 2020, doi: 10.14710/buloma.v9i1.23924.

J. L. Zhao et al., “Inter-rater and Intra-rater Reliability of the Chinese Version of the Action Research Arm Test in People With Stroke,” Front Neurol, vol. 10, no. MAY, 2019, doi: 10.3389/FNEUR.2019.00540.

T. K. Koo and M. Y. Li, “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research,” J Chiropr Med, vol. 15, no. 2, pp. 155–163, Jun. 2016, doi: 10.1016/J.JCM.2016.02.012.

V. S. Senthil Kumar and S. Shahraz, “Intraclass correlation for reliability assessment: the introduction of a validated program in SAS (ICC6),” Health Serv Outcomes Res Methodol, pp. 1–13, Mar. 2023, doi: 10.1007/S10742-023-00299-X/TABLES/11.

G. Perinetti, “StaTips Part IV: Selection, interpretation and reporting of the intraclass correlation coefficient,” South Eur. J. Orthod. Dentofac. Res. SEJODR, vol. 5, no. 1, pp. 3–5, May 2018, doi: 10.5937/SEJODR5-17434.