UX (USER EXPERIENCE) EVALUATION OF THE OPENLEARNING SYSTEM AT UNIVERSITAS MUHAMMADIYAH SURAKARTA USING HEURISTIC EVALUATION AND USABILITY TESTING

Afifah Ghaisani Imana; Yusuf Sulistyo  Nugroho

doi:10.52436/1.jutif.2023.4.4.824

Afifah Ghaisani Imana Informatics, Faculty of Communication and Informatics, Universitas Muhammadiyah Surakarta, Indonesia
Yusuf Sulistyo Nugroho Informatics, Faculty of Communication and Informatics, Universitas Muhammadiyah Surakarta, Indonesia

DOI: https://doi.org/10.52436/1.jutif.2023.4.4.824

Keywords: heuristic evaluation, learning management system, openlearning, usability testing, user experience

Abstract

Since the COVID-19 pandemic in 2020, the learning process at the University of Muhammadiyah Surakarta (UMS) has been carried out online using the OpenLearning platform. The use of OpenLearning media has provided an easy learning process for both lecturers and students. Although there are many conveniences and benefits offered, usability problems still arise, for example, there is no distinction between courses that have ended and those that are still active, the "Edit Page Header" feature on student users, and misperceptions in the progress bar in the Course menu. In addition, previous research has evaluated the UMS OpenLearning platform using the System Usability Scale (SUS) technique and scored 61 (grade scale D) and indicates that the system is well received. To identify more detailed problems related to user experience (UX) for the OpenLearning system applied at UMS, it is necessary to evaluate the UX with the other approaches. In this study, the Heuristic Evaluation method based on 10 principles of Heuristic and Usability Testing based on 5 Usability criteria are implemented to evaluate. The results of this study are a system prototype according to the results of heuristic evaluation. The prototype was evaluated using SUS method on 380 respondents. The SUS calculation yields a value of 69, which indicates that the system is classified to the grade scale category D. This insignificant increase in the SUS score indicates that the UX improvements have not been optimally related to usability due to some factors such as the complexity of the system when used, inconsistencies, and the need for users to adapt to using the system. This finding can be considered by the system development team to fix the weaknesses so as to create an online learning platform with a better UX.

Downloads

Download data is not yet available.

References

B. M. Batubara, “The Problems of the World of Education in the Middle of the Covid-19 Pandemic,” Budapest Int. Res. Critics Inst. Humanit. Soc. Sci., vol. 4, no. 1, pp. 450–457, 2021.

A. Abidah, H. N. Hidaayatullaah, R. M. Simamora, D. Fehabutar, and L. Mutakinati, “The Impact of Covid-19 to Indonesian Education and Its Relation to the Philosophy of ‘Merdeka Belajar,’” Stud. Philos. Sci. Educ., vol. 1, no. 1, pp. 38–49, 2020.

A. Fatmawati, “Evaluasi Usability pada Learning Management System OpenLearning Menggunakan System Usability Scale,” INOVTEK Polbeng - Seri Inform., vol. 6, no. 1, p. 120, 2021.

M. I. S. B. Khairat, Y. Priyadi, and M. Adrian, “Usability Measurement in User Interface Design Using Heuristic Evaluation Severity Rating (Case Study: Mobile TA Application based on MVVM),” 2022 IEEE 12th Annu. Comput. Commun. Work. Conf. CCWC 2022, pp. 974–979, 2022.

J. S. Dumas and J. E. Fox, “Usability Testing: A Practitioner’s Guide to Evaluating the User Experience,” Human-Computer Interact. Handb. Fundam. Evol. Technol. Emerg. Appl. Third Ed., pp. 1221–1241, 2012.

I. Fauziah and N. R. Nurwulan, “Usability Evaluation of Google Classroom for Elementary School Students,” ACM Int. Conf. Proceeding Ser., pp. 11–15, Sep. 2021.

D. Pal and V. Vanijja, “Perceived usability evaluation of Microsoft Teams as an online learning platform during COVID-19 using system usability scale and technology acceptance model in India,” Child. Youth Serv. Rev., vol. 119, p. 105535, Dec. 2020.

A. F. Rahmawati, T. Wahyuningrum, A. C. Wardhana, A. Septiari, and L. Afuan, “User Experience Evaluation Using Integration of Remote Usability Testing and Usability Evaluation Questionnaire Method,” Proc. - 2022 IEEE Int. Conf. Cybern. Comput. Intell. Cybern. 2022, pp. 40–45, 2022.

U. Müssener et al., “Development of an intervention targeting multiple health behaviors among high school students: Participatory design study using heuristic evaluation and usability testing,” JMIR mHealth uHealth, vol. 8, no. 10, pp. 1–12, 2020.

T. Wahyuningrum, C. Kartiko, and A. C. Wardhana, “Exploring e-Commerce Usability by Heuristic Evaluation as a Compelement of System Usability Scale,” 2020 Int. Conf. Adv. Data Sci. E-Learning Inf. Syst. ICADEIS 2020, Oct. 2020.

A. Petrovčič, A. Rogelj, and V. Dolničar, “Smart but not adapted enough: Heuristic evaluation of smartphone launchers with an adapted interface and assistive technologies for older adults,” Comput. Human Behav., vol. 79, pp. 123–136, 2018.

J. R. Lewis and J. Sauro, “Item Benchmarks for the System Usability Scale,” J. Usability Stud., vol. 13, no. May 2018, pp. 158–167, 2018.

ISO, “Ergonomics of human-system interaction - Part 11: Usability: Definitions and Concepts,” Iso 9241-11:2018(E), 2018. https://infostore.saiglobal.com/preview/is/en/2018/i.s.eniso9241-11-2018.pdf?sku=1980667%0Ahttps://www.sis.se/api/document/preview/80003410/ (accessed Oct. 07, 2022).

J. Nielsen, “Finding usability problems through heuristic evaluation,” Conf. Hum. Factors Comput. Syst. - Proc., pp. 373–380, 1992.

J. Nielsen and R. Molich, “Heuristic evaluation of user interfaces,” Conf. Hum. Factors Comput. Syst. - Proc., no. April, pp. 249–256, 1990.

K. Kous, M. Pušnik, M. Heričko, and G. Polančič, “Usability evaluation of a library website with different end user groups,” J. Librariansh. Inf. Sci., vol. 52, no. 1, pp. 75–90, Mar. 2020.

J. Brooke, “SUS: A ‘Quick and Dirty’ Usability Scale,” Usability Eval. Ind., no. June, pp. 207–212, 1996.

G. T. Kurniaji, M. F. Eza ’Ammar, and Y. S. Nugroho, “The Development of UMS Building Catalogue Information System,” Ultim. J. Tek. Inform., vol. 14, no. 1, pp. 13–19, 2022.

A. Kaya, R. Ozturk, and C. Altin Gumussoy, “Usability Measurement of Mobile Applications with System Usability Scale (SUS),” pp. 389–400, 2019.

J. Nielsen, “How to Conduct a Heuristic Evaluation,” Nov. 01, 1994. https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/ (accessed Nov. 01, 2022).

J. Nielsen, “Why You Only Need to Test with 5 Users,” Mar. 18, 2000. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ (accessed Nov. 01, 2022).

F. Fatihahsari and C. Darujati, “Analisis Usability Mobile Apps Edlink dengan Menggunakan Heuristic Evaluation,” Sistemasi, vol. 10, no. 2, p. 404, 2021.

H. Nadhirah, N. Hendrakusma Wardani, and K. Candra Brata, “Evaluasi Usability dan Perbaikan Desain Website Dinas Pendidikan Kota Malang menggunakan Metode Heuristic Evaluation dengan Prinsip Usability G-Quality,” J. Pengemb. Teknol. Inf. dan Ilmu Komput., vol. 3, no. 6, pp. 6115–6124, 2019.

J. Nielsen, “Severity Ratings for Usability Problems,” Nov. 01, 1994. https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/ (accessed Nov. 02, 2022).

D. Sulistyaningrum, H. Tolle, and R. I. Rokhmawati, “Evaluasi dan Perbaikan User Experience Pada Aplikasi Mobile UB TV Dengan Metode Heuristic Evaluation dan Usability Testing,” J. Pengemb. Teknol. Inf. dan Ilmu Komput., vol. 3, no. 7, pp. 6294–6302, 2019, Accessed: Sep. 25, 2022.

J. J. Tejada, J. Raymond, and B. Punzalan, “On the Misuse of Slovin’s Formula,” Philipp. Stat., vol. 61, no. 1, p. 8, 2012.