Comparative Evaluation of Retrieval Augmented Generation Based Large Language Model with Formative Feedback and Traditional Instruction in Learning Fraction
DOI:
https://doi.org/10.52436/1.jutif.2026.7.5.5847Keywords:
Formative Feedback, Fraction Learning, Intelligent Tutoring System, Large Language Models, Retrieval Augmented GenerationAbstract
Learning fractions poses challenges for students, and while traditional one-on-one tutoring is effective, it faces human resource limitations. While generative artificial intelligence (AI) offers scalable personalized support, large language models (LLMs) risk generating hallucinations, leading students to misconceptions. To address these challenges, this study evaluates the effect of a Retrieval-Augmented Generation (RAG) based Intelligent Tutoring System (ITS) integrated with formative feedback on students' learning performance in fraction arithmetic, comparing it to traditional instruction. A quasi-experimental study was conducted with 56 7th grader students, divided into an experimental group (N=26) utilizing the ITS and a control group (N=30) receiving regular classroom instruction. The ITS employed GPT-4o and RAG grounded with fraction educational content to deliver personalized, step-by-step formative feedback based on the Mason and Bruning framework. Students’ learning performance was measured through pre and post-test and were analyzed using ANCOVA. The experimental group outperformed the control group with higher learning gain of 47.02% compared to the control group’s 31.54%. ANCOVA results confirmed the significant main effect (p = 0.002) suggesting that the RAG-based tutor improved the mathematical comprehension and reduced knowledge variance among students. These results underscore the grounded LLM integrated with formative feedback demonstrates potential to improve students learning performance. This study provides a comparative evaluation of RAG-based ITS integrated with formative feedback and traditional classroom instruction in improving student learning performance in fraction. Furthermore, this study advances AI-supported tutoring by introducing an LLM-based ITS that combines RAG-grounded generation, formative feedback, and LLM-based judgment for dynamic and adaptive student modelling.
Downloads
References
M. Hariyani, T. Herman, D. Suryadi, and S. Prabawanto, “Exploration of Student Learning Obstacles in Solving Fraction Problems in Elementary School,” Int. J. Educ. Method., vol. 8, no. 3, pp. 505–515, Aug. 2022, doi: 10.12973/ijem.8.3.505.
S. C. Shih, C. C. Chang, B. C. Kuo, and Y. H. Huang, “Mathematics intelligent tutoring system for learning multiplication and division of fractions based on diagnostic teaching,” Educ. Inf. Technol. (Dordr)., vol. 28, no. 7, pp. 9189–9210, Jul. 2023, doi: 10.1007/s10639-022-11553-z.
M. Hariyani, H. Herawati, M. Andriani, and S. Suherman, “Students’ learning obstacles in understanding of fraction concept during online learning,” Journal of Didactic Mathematics, vol. 4, no. 2, pp. 58–64, Aug. 2023, doi: 10.34007/jdm.v4i2.1849.
A. M. Jarrah, Y. Wardat, and S. Gningue, “Misconception on addition and subtraction of fractions in seventh-grade middle school students,” Eurasia Journal of Mathematics, Science and Technology Education, vol. 18, no. 6, p. em2115, May 2022, doi: 10.29333/ejmste/12070.
J. A. Rodríguez‐Martínez, J. A. González‐Calero, J. del Olmo‐Muñoz, D. Arnau, and S. Tirado‐Olivares, “Building personalised homework from a learning analytics based formative assessment: Effect on fifth‐grade students’ understanding of fractions,” British Journal of Educational Technology, vol. 54, no. 1, pp. 76–97, Jan. 2023, doi: 10.1111/bjet.13292.
F. Weber, T. Wambsganss, and M. Söllner, “Enhancing legal writing skills: The impact of formative feedback in a hybrid intelligence learning environment,” British Journal of Educational Technology, vol. 56, no. 2, pp. 650–677, Mar. 2025, doi: 10.1111/bjet.13529.
S. Steinert, K. E. Avila, S. Ruzika, J. Kuhn, and S. Küchemann, “Harnessing large language models to develop research-based learning assistants for formative feedback,” Dec. 01, 2024, Springer. doi: 10.1186/s40561-024-00354-1.
V. J. Shute, “Focus on Formative Feedback,” Rev. Educ. Res., vol. 1, no. 78, pp. 153–189, Mar. 2007, doi: https://doi.org/10.3102/0034654307313795.
B. S. Bloom, “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring,” Educational Researcher, vol. 13, pp. 4–16, Jun. 1984, doi: https://doi.org/10.3102/0013189X013006004.
Z. Levonian et al., “Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference,” Oct. 2023, doi: doi.org/10.5281/zenodo.12729824.
Z. Liu, P. Agrawal, S. Singhal, V. Madaan, M. Kumar, and P. K. Verma, “LPITutor: An LLM based personalized intelligent tutoring system using RAG and prompt engineering,” PeerJ Comput. Sci., vol. 11, 2025, doi: 10.7717/peerj-cs.2991.
R. Kabir and F. Lin, “An LLM-Powered Adaptive Practicing System,” 2023. Accessed: Mar. 20, 2026. [Online]. Available: http://ceur-ws.org
Y. Lee, “Developing a computer-based tutor utilizing Generative Artificial Intelligence (GAI) and Retrieval-Augmented Generation (RAG),” Educ. Inf. Technol. (Dordr)., Apr. 2024, doi: 10.1007/s10639-024-13129-5.
F. Zhang et al., “Math-LLMs: AI Cyberinfrastructure with Pre-trained Transformers for Math Education,” Int. J. Artif. Intell. Educ., 2024, doi: 10.1007/s40593-024-00416-y.
Y. Fang, B. He, Z. Liu, S. Liu, Z. Yan, and J. Sun, “Evaluating the Design Features of an Intelligent Tutoring System for Advanced Mathematics Learning,” Jul. 2024, pp. 338–350. doi: https://doi.org/10.1007/978-3-031-64302-6_24.
H. S. Nwana, “Intelligent Tutoring Systems: an overview,” Artif. Intell. Rev., vol. 4, pp. 251–277, 1990.
C. Papakostas, C. Troussas, A. Krouska, and C. Sgouropoulou, “A Rule-Based Chatbot Offering Personalized Guidance in Computer Programming Education,” 2024, pp. 253–264. doi: 10.1007/978-3-031-63031-6_22.
K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval Augmentation Reduces Hallucination in Conversation,” Apr. 2021, [Online]. Available: http://arxiv.org/abs/2104.07567
A. Brown, M. Roman, and B. Devereux, “A Systematic Literature Review of Retrieval-Augmented Generation: Techniques, Metrics, and Challenges,” Big Data and Cognitive Computing, vol. 9, no. 12, p. 320, Dec. 2025, doi: 10.3390/bdcc9120320.
A. Gupta, J. Reddig, T. Calo, D. Weitekamp, and C. J. MacLellan, “Beyond Final Answers: Evaluating Large Language Models for Math Tutoring,” Feb. 2025, [Online]. Available: http://arxiv.org/abs/2503.16460
M. Kaplar et al., “SPARK_AI: A Prompt-Orchestrated Architecture for Stateful, Process-Oriented Reasoning with Large Language Models,” Informatics, vol. 13, no. 4, p. 63, Apr. 2026, doi: 10.3390/informatics13040063.
H. Li et al., “Who Should Be My Tutor? Analyzing the Interactive Effects of Automated Text Personality Styles Between Middle School Students and a Mathematics Chatbot,” in 15th International Conference on Learning Analytics and Knowledge, LAK 2025, Association for Computing Machinery, Inc, Mar. 2025, pp. 910–917. doi: 10.1145/3706468.3706537.
Z. A. Pardos and S. Bhandari, “ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills,” PLoS One, vol. 19, no. 5 May, May 2024, doi: 10.1371/journal.pone.0304013.
W. Dai et al., “Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT,” in 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), IEEE, Jul. 2023, pp. 323–325. doi: 10.1109/ICALT58122.2023.00100.
S. Baral et al., “Automated Feedback in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses,” Oct. 2024, [Online]. Available: http://arxiv.org/abs/2411.08910
K. Enevoldsen et al., “MMTEB: Massive Multilingual Text Embedding Benchmark,” Nov. 2025, [Online]. Available: http://arxiv.org/abs/2502.13595
J. Macina et al., “MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems,” in Findings of the Association for Computational Linguistics: EMNLP 2023, Stroudsburg, PA, USA: Association for Computational Linguistics, 2023, pp. 5602–5621. doi: 10.18653/v1/2023.findings-emnlp.372.
N. Singh, V. K. Gunjan, and J. M. Zurada, Cognitive Tutor Custom-Tailored Pedagogical Approach. in Advanced Technologies and Societal Change. Singapore: Springer Nature Singapore, 2022. doi: 10.1007/978-981-19-5197-8.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Ida Bagus Indrabudhi Kusuma, Dade Nurjanah

This work is licensed under a Creative Commons Attribution 4.0 International License.

</a



