Random Forest Machine Learning Analysis of Generative AI’s Impact on Learning Effectiveness in Indonesian Higher Education

Sulfikar Sallu; Hendriadi Hendriadi

doi:10.52436/1.jutif.2025.6.5.5365

Authors

Sulfikar Sallu Fakultas Keguruan dan Ilmu Pendidikan, Universitas Sulawesi Tenggara, Indonesia
Hendriadi Fakultas Ekonomi dan Bisnis, Universitas Sulawesi Tenggara, Indonesia

DOI:

https://doi.org/10.52436/1.jutif.2025.6.5.5365

Keywords:

Generative AI, Higher Education, Indonesia, Informatics, Learning Effectiveness, Machine Learning, Random Forest

Abstract

Generative Artificial Intelligence (GenAI) has rapidly penetrated Indonesian higher education, creating opportunities for learning innovation while raising concerns about effectiveness and academic integrity. This study develops a machine learning–based quantitative model to analyze the impact of GenAI usage on learning effectiveness, with a particular focus on Informatics students as key digital literacy stakeholders. Data were collected from a simulated survey of 300 students, covering demographics, GPA, exam scores, GenAI usage patterns, digital literacy, motivation, self-efficacy, academic integrity, and institutional support. Preprocessing steps included normalization of continuous variables, one-hot encoding of categorical variables, and feature selection using Recursive Feature Elimination (RFE). Six machine learning algorithms—Logistic Regression, Decision Tree, Random Forest, Support Vector Machine (SVM), XGBoost, and Artificial Neural Network—were compared to identify the best predictive model. Results show that Random Forest achieved the highest performance, with 87% accuracy and an AUC greater than 0.90, significantly outperforming other algorithms. The most influential predictors were digital literacy, institutional policies, and frequency of GenAI usage, while demographic variables contributed minimally. These findings suggest that GenAI can enhance learning effectiveness in Informatics education when supported by critical digital literacy and ethical awareness. The novelty of this study lies in integrating survey-based educational data with Random Forest machine learning to empirically model GenAI’s role in Indonesian higher education. The results provide practical implications for policymakers, educators, and institutions to design AI-integrated learning strategies that maximize innovation while safeguarding academic integrity.

Downloads

Download data is not yet available.

References

A. Henadirage and N. Gunarathne, “Barriers to and opportunities for the adoption of generative artificial intelligence in higher education in the global south: Insights from Sri Lanka,” Int. J. Artif. Intell. Educ., vol. 35, no. 1, pp. 245–281, 2025, doi: 10.1007/s40593-024-00439-5.

A. Arista, L. Shuib, and M. A. Ismail, “Systematicliterature review and bibliometric analysis on ethical policies for generative artificial intelligence (GAI) in higher education institutions (HEIs),” in 2024 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS), 2024, pp. 454–459. doi: 10.1109/ICIMCIS63449.2024.10956736.

M. E. S. Simaremare, C. Pardede, I. N. I. Tampubolon, D. A. Simangunsong, and P. E. Manurung, “The penetration of generative AI in higher education: A survey,” in 2024 IEEE Integrated STEM Education Conference (ISEC), Princeton: IEEE Xplore, 2024, pp. 1–5. doi: 10.1109/ISEC61299.2024.10664825.

M. Belkina et al., “Implementing generative AI (GenAI) in higher education: A systematic review of case studies,” Comput. Educ. Artif. Intell., vol. 8, no. 2025, p. 100407, 2025, doi: 10.1016/j.caeai.2025.100407.

A. E. E. Sobaih and A. E. A. Elnasr, “Battle of AI chatbots: Graduate students’ perceptions of ChatGPT versus Gemini for learning purposes in Egyptian higher education,” J. Appl. Learn. Teach., vol. 8, no. 1, pp. 128–142, Jan. 2025, doi: 10.37074/jalt.2025.8.1.7.

K. Bista and R. Bista, “Leveraging AI tools in academic writing: Insights from doctoral students on benefits and challenges,” Am. J. STEM Educ. Issues Perspect., vol. 6, no. 2025, pp. 32–47, 2025, doi: 10.32674/9m8dq081.

C. Boyle, “ChatGPT, gemini, & copilot: Using generative AI as a tool for information literacy instruction,” Ref. Libr., vol. 66, no. 1–2, pp. 13–29, Apr. 2025, doi: 10.1080/02763877.2025.2465416.

X. Han, H. Peng, and M. Liu, “The impact of GenAI on learning outcomes: A systematic review and meta-analysis of experimental studies,” Educ. Res. Rev., vol. 48, no. 2025, p. 100714, 2025, doi: 10.1016/j.edurev.2025.100714.

A. Yusuf, N. Pervin, and M. Román-González, “Generative AI and the future of higher education: A threat to academic integrity or reformation? Evidence from multicultural perspectives,” Int. J. Educ. Technol. High. Educ., vol. 21, no. 2024, pp. 1–29, 2024, doi: 10.1186/s41239-024-00453-6.

I. Isabella, A. Alfitri, A. Saptawan, N. Nengyanti, and T. Baharuddin, “Empowering digital citizenship in Indonesia: Navigating urgent digital literacy challenges for effective digital governance,” J. Gov. Public Policy, vol. 11, no. 2, pp. 142–155, 2024, doi: 10.18196/jgpp.v11i2.19258.

E. C. Wang, “Technology-driven education reform in Indonesia,” OliverWyman, vol. 1, no. 1, pp. 18–24, 2022.

K. Saddhono, L. Judijanto, T. Nuryanto, M. Y. Anis, J. Nasution, and S. Sallu, “The AI based design layout for exploring the reader experience through technical interaction for learning literature,” in 2024 4th International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida: IEEE Xplore, 2024, pp. 1489–1494. doi: 10.1109/ICACITE60783.2024.10617284.

K. Daniel, M. M. Msambwa, and Z. Wen, “Can generative AI revolutionise academic skills development in higher education? A systematic literature review,” Eur. J. Educ., vol. 60, no. 1, p. e70036, 2025, doi: 10.1111/ejed.70036.

M. Ala, S. Shahid, S. Mahmud, S. Mohyuddin, and K. Kaur, “Perceived influence of GenAI on student engagement in online higher education,” J. Appl. Learn. Teach., vol. 8, no. 2, pp. 25–34, 2025, doi: 10.37074/jalt.2025.8.2.16.

F. Wang, N. Li, A. C. K. Cheung, and G. K. W. Wong, “In GenAI we trust: An investigation of university students’ reliance on and resistance to generative AI in language learning,” Int. J. Educ. Technol. High. Educ., vol. 22, no. 1, p. 59, 2025, doi: 10.1186/s41239-025-00547-9.

L. Raitskaya and E. Tikhonova, “Enhancing critical thinking skills in ChatGPT-human interaction: A scoping review,” J. Lang. Educ., vol. 11, no. 2, pp. 5–19, 2025, doi: 10.17323/jle.2025.27387.

J. Gu and Z. Yan, “Effects of genAI interventions on student academic performance: A meta-analysis,” J. Educ. Comput. Res., vol. 63, no. 6, pp. 1460–1492, 2025, doi: 10.1177/07356331251349620.

X. O’Dea and M. O’Dea, “Is artificial intelligence really the next big thing in learning and teaching in higher education?: A conceptual paper,” J. Univ. Teach. Learn. Pract., vol. 20, no. 5, pp. 1–17, Aug. 2023, doi: 10.53761/1.20.5.05.

S. Feng and N. Law, “Mapping artificial intelligence in education research: A network‐based keyword analysis,” Int. J. Artif. Intell. Educ., vol. 31, no. 2, pp. 277–303, 2021, doi: 10.1007/s40593-021-00244-4.

Suwendi, Mesraini, C. B. Gama, H. Rahman, T. Luhuringbudi, and M. Masrom, “Adoption of artificial intelligence and digital resources among academicians of islamic higher education institutions in Indonesia,” J. Online Inform., vol. 10, no. 1, pp. 42–52, 2025, doi: 10.15575/join.v10i1.1549.

D. Wong-A-Foe, “Navigating the implications of AI in Indonesian education: Tutors, governance, and ethical perspectives,” in Data Science and Artificial Intelligence, C. Anutariya and M. M. Bonsangue, Eds., Singapore: Springer Nature Singapore, 2023, pp. 349–360. doi: 10.1007/978-981-99-7969-1_26.

S. D. Datta, M. Islam, M. H. R. Sobuz, S. Ahmed, and M. Kar, “Artificial intelligence and machine learning applications in the project lifecycle of the construction industry: A comprehensive review,” Heliyon, vol. 10, no. 5, p. e26888, 2024, doi: 10.1016/j.heliyon.2024.e26888.

S. F. Ahmed et al., “Deep learning modelling techniques: Current progress, applications, advantages, and challenges,” Artif. Intell. Rev., vol. 56, no. 11, pp. 13521–13617, 2023, doi: 10.1007/s10462-023-10466-8.

I. Gligorea, M. Cioca, R. Oancea, A.-T. Gorski, H. Gorski, and P. Tudorache, “Adaptive learning using artificial intelligence in e-learning: A literature review,” Educ. Sci., vol. 13, no. 12, pp. 1–27, 2023, doi: 10.3390/educsci13121216.

D. Patil, N. Rane, N. Rane, and P. Desai, “Machine learning and deep learning: Methods, techniques, applications, challenges, and future research opportunities,” in Trustworthy Artificial Intelligence in Industry and Society, 1st ed., London: Deep Science Publishing, 2024, pp. 28–81. doi: 10.70593/978-81-981367-4-9_2.

M. Rezaei, “Artificial intelligence in knowledge management: Identifying and addressing the key implementation challenges,” Technol. Forecast. Soc. Change, vol. 217, p. 124183, 2025, doi: 10.1016/j.techfore.2025.124183.

M. F. Zhang, J. F. Dawson, and R. B. Kline, “Evaluating the use of covariance-based structural equation modelling with reflective measurement in organizational and management research: A review and recommendations for best practice,” Br. J. Manag., vol. 32, no. 2, pp. 257–272, 2021, doi: 10.1111/1467-8551.12415.

T.-C. Hsu and M.-S. Chen, “Effects of students using different learning approaches for learning computational thinking and AI applications,” Educ. Inf. Technol., vol. 30, no. 6, pp. 7549–7571, 2025, doi: 10.1007/s10639-024-13116-w.

F. Trujillo, M. Pozo, and G. Suntaxi, “Artificial intelligence in education: A systematic literature review of machine learning approaches in student career prediction,” J. Technol. Sci. Educ., vol. 15, no. 1, pp. 162–185, 2025, doi: 10.3926/JOTSE.3124.

M. Zaim et al., “AI-powered EFL pedagogy: Integrating generative AI into university teaching preparation through UTAUT and activity theory,” Comput. Educ. Artif. Intell., vol. 7, no. 2024, p. 100335, 2024, doi: 10.1016/j.caeai.2024.100335.