the ENHANCE OBJECT TRACKING ON AUGMENTED REALITY USING HYBRID CONVOLUTIONAL NEURAL NETWORK AND FAST CORNER DETECTION

nurhadi nurhadi; Eko Arip  Winanto; Saparudin

doi:10.52436/1.jutif.2025.6.1.4257

nurhadi nurhadi Department of Informatics, Universitas Dinamika Bangsa, Jambi, Indonesia
Eko Arip Winanto Computer Science, Universiti Teknologi Malaysia, Johor, Malaysia
Saparudin Department of Informatics Engineering, Telkom University, Indonesia

DOI: https://doi.org/10.52436/1.jutif.2025.6.1.4257

Keywords: AR, Object tracking, CNN, Fast corner detection, Markerless

Abstract

Markerless augmented reality (AR) is utilized in applications that do not require anchoring to the real world and do not require the use of physical markers (fiducial markers). Augmented object displays not only float but also allow for the automatic placement of 3D augmented reality objects on flat surfaces to enhance realism in real time. There are two challenges that need to be addressed in Markerless AR systems: object tracking and registration, as well as the influence of light intensity. Therefore, the objective of this research is to propose the use of Convolutional Neural Networks (CNN) and Features from Accelerated Segment Test (FAST) corner detection for tracking or detecting objects in markerless augmented reality systems. Testing was conducted using three epoch schemes: 10, 50, and 100. The test results were measured using several parameters, including the execution time, testing loss, and testing accuracy. The test results indicated an improvement in the performance of the tested object detection. The accuracy testing results of using the CNN and FAST corner detection methods were superior to those of the CNN-only method and FAST corner detection alone, reaching 98%. However, this method increases the processing time for object detection. Thus, the processing time of the CNN without FAST corner detection was faster.

Downloads

Download data is not yet available.

References

C. Gsaxner et al., “Augmented Reality for Head and Neck Carcinoma Imaging: Description and Feasibility of an Instant Calibration, Markerless Approach,” Comput. Methods Programs Biomed., vol. 200, p. 105854, 2021, doi: 10.1016/j.cmpb.2020.105854.

W. Li, J. Wang, S. Jiao, M. Wang, and S. Li, “Fully convolutional network-based registration for augmented assembly systems,” J. Manuf. Syst., no. December, 2021, doi: 10.1016/j.jmsy.2020.12.017.

A. Brunzini, M. Ciccarelli, M. Sartini, G. Menchi, A. Papetti, and M. Germani, "A novel approach to use marker-less mixed reality applications with in-motion systems," in Advances on Mechanics, Design Engineering and Manufacturing IV, S. Gerbino, A. Lanzotti, M. Martorelli, R. M. Buil, C. Rizzi, and L. Roucoules, Eds. Cham: Springer International Publishing, 2023, pp. 1401–1412. doi: 10.1007/978-3-031-15928-2_122.

S. Boonbrahm, P. Boonbrahm, and C. Kaewrat, “The use of marker-based augmented reality in space measurement,” Procedia Manuf., vol. 42, no. 2019, pp. 337–343, 2020, doi: 10.1016/j.promfg.2020.02.081.

X. Hu, F. R. y Baena, and F. Cutolo, "Head-mounted augmented reality platform for markerless orthopaedic navigation," IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 2, pp. 910–921, 2022. doi: 10.1109/JBHI.2021.3088442

T. Jia, Z. A. Taylor, and X. Chen, "Long term and robust 6DoF motion tracking for highly dynamic stereo endoscopy videos," Computerized Medical Imaging and Graphics, vol. 94, p. 101995, 2021. doi: 10.1016/j.compmedimag.2021.101995.

C.-H. Chu, C.-Y. Weng, and Y.-T. Chen, "Enhancing manual inspection in semiconductor manufacturing with integrated augmented reality solutions," Journal of Manufacturing Systems, vol. 77, pp. 933–945, 2024. doi: 10.1016/j.jmsy.2024.10.028.

M. Venkatesan et al., “Virtual and augmented reality for biomedical applications,” Cell Reports Med., vol. 2, no. 7, pp. 1–13, 2021, doi: 10.1016/j.xcrm.2021.100348.

J. M. Runji, Y.-J. Lee, and C.-H. Chu, "Systematic literature review on augmented reality-based maintenance applications in manufacturing centered on operator needs," International Journal of Precision Engineering and Manufacturing-Green Technology, vol. 10, no. 2, pp. 567–585, Mar. 2023. doi: 10.1007/s40684-022-00444-w.

J. Angelopoulos and D. Mourtzis, "An intelligent product service system for adaptive maintenance of engineered-to-order manufacturing equipment assisted by augmented reality," Applied Sciences, vol. 12, no. 11, Article no. 5349, 2022. doi: 10.3390/app12115349.

J. Song and J. Kook, "Visual SLAM based spatial recognition and visualization method for mobile AR systems," Applied System Innovation, vol. 5, no. 1, Article no. 11, 2022. doi: 10.3390/asi5010011.

M. Wang, C. Sun, and A. Sowmya, "Efficient corner detection based on corner enhancement filters," Digital Signal Processing, vol. 122, p. 103364, 2022. doi: 10.1016/j.dsp.2021.103364.(E. Rosten and T. Drummond, “Machine Learning for High-Speed Corner Detection,” in Computer Vision -- ECCV 2006, 2006, pp. 430–443.)

N. Nurhadi, E. A. Winanto, and S. Saparudin, “Enhance Object Tracking on Augmented Reality Markerless using FAST Corner Detection,” 2021 IEEE 5th Int. Conf. Inf. Technol. Inf. Syst. Electr. Eng., pp. 66–70, 2022.

V. Arulkumar, S. Jaya Prakash, E. K. Subramanian, and N. Thangadurai, "An intelligent face detection by corner detection using special morphological masking system and fast algorithm," in Proc. 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 2021, pp. 1556-1561. doi: 10.1109/ICOSEC51865.2021.9591857.

S. Chakraborty and L. Dey, "Virtual and Augmented Reality with Embedded Systems," in Computing for Data Analysis: Theory and Practices, Singapore: Springer Nature Singapore, 2023, pp. 75-96. doi: 10.1007/978-981-19-8004-6_4.

X. Shi, W. Xu, and J. Wang, "An improved feature point-based LK optical flow algorithm for robotic arm control," in Eighth International Symposium on Advances in Electrical, Electronics, and Computer Engineering (ISAEECE 2023), Shubin Yan and Pavel Loskot, Eds., SPIE, 2023, vol. 12704, pp. 127043G. doi: 10.1117/12.2680052.

J. Li, B. Wang, S. Zhu, X. Cao, F. Zhong, W. Chen, T. Li, J. Gu, and X. Qin, "BCOT: A Markerless High-Precision 3D Object Tracking Benchmark," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 6697-6706.

Y. Ghasemi, H. Jeong, S. H. Choi, K.-B. Park, and J. Y. Lee, "Deep learning-based object detection in augmented reality: A systematic review," Computers in Industry, vol. 139, 2022, p. 103661. doi.org/10.1016/j.compind.2022.103661.

K. Mahdikhanlou and H. Ebrahimnezhad, "3D object deforming and manipulating through dynamic hand gestures," Entertainment Computing, vol. 43, 2022, p. 100507. [Online]. Available: https://doi.org/10.1016/j.entcom.2022.100507.

A. Gaurav, B. B. Gupta, and P. K. Panigrahi, "A comprehensive survey on machine learning approaches for malware detection in IoT-based enterprise information system," Enterprise Information Systems, vol. 17, no. 3, p. 2023764, 2023. [Online]. Available: https://doi.org/10.1080/17517575.2021.2023764.

M. M. Srikantamurthy, V. P. S. Rallabandi, D. B. Dudekula, S. Natarajan, and J. Park, "Classification of benign and malignant subtypes of breast cancer histopathology imaging using hybrid CNN-LSTM based transfer learning," BMC Medical Imaging, vol. 23, no. 1, p. 19, 2023., doi.org/10.1186/s12880-023-00964-0.

G. Apostolopoulos, D. Andronas, N. Fourtakas, and S. Makris, “Operator training framework for hybrid environments: An Augmented Reality module using machine learning object recognition,” Procedia CIRP, vol. 106, pp. 102–107, 2022, doi: 10.1016/j.procir.2022.02.162.

A. Rahman et al., “An integrated framework of sensing, machine learning, and augmented reality for aquaculture prawn farm management,” Aquac. Eng., vol. 95, no. July, p. 102192, 2021, doi: 10.1016/j.aquaeng.2021.102192.

K.-B. Park, S. H. Choi, and J. Y. Lee, "Self-training based augmented reality for robust 3D object registration and task assistance," Expert Systems with Applications, vol. 238, Part F, p. 122331, 2024.

S. Raj, L. R. D. Murthy, T. A. Shanmugam, G. Kumar, A. Chakrabarti, and P. Biswas, "Augmented reality and deep learning based system for assisting assembly process," Journal on Multimodal User Interfaces, vol. 18, no. 1, pp. 119-133, 2024. [Online]. Available: https://doi.org/10.1007/s12193-023-00428-3.