Bidirectional Cross-Attention and Uncertainty-Aware Ensemble for High-Precision Brain Tumor Classification on MRI Images

Gunawan Cholis Saputra; Muhammad Yuwanandra Risdyaksa

doi:10.52436/1.jutif.2026.7.2.5494

Authors

Gunawan Cholis Saputra Information System, Universitas Dian Nuswantoro, Indonesia
Muhammad Yuwanandra Risdyaksa Information System, Universitas Dian Nuswantoro, Indonesia

DOI:

https://doi.org/10.52436/1.jutif.2026.7.2.5494

Keywords:

Bidirectional Cross-Attention, Brain Tumor Classification, Deep Learning, EfficientNet, MRI Analysis, Swin Transformer, Uncertainty Quantification

Abstract

Accurate brain tumor diagnosis via Magnetic Resonance Imaging (MRI) is vital for effective neuro-oncological treatment. Although CNNs are widely regarded as the benchmark for local texture extraction, they frequently exhibit limitations in modeling long-distance global dependencies effectively. In contrast, Vision Transformers (ViTs), particularly the Swin variant, demonstrate superior capability in capturing global semantic context yet often fail to preserve the fine local granularity needed to delineate tumor boundaries significantly. To bridge this gap, we propose Bi-CA-UAE, a hybrid framework integrating Swin Transformer and EfficientNet-V2 through a novel Bidirectional Cross-Attention mechanism. Unlike static ensembles, our method enables dynamic information exchange between global and local feature maps before classification. Furthermore, we introduce an Uncertainty-Aware Gating Network to adaptively weigh each branch based on prediction confidence, reducing false positives in ambiguous cases. Validated on a multi-class MRI dataset of 7,023 images, the model achieved 99.85% accuracy and an Expected Calibration Error (ECE) of 0.02, matching the strongest baseline (Swin Transformer) while demonstrating superior training stability and calibration. Unlike naive concatenation ensembles that suffered from overfitting and performance degradation in later training stages, Bi-CA-UAE maintained robust peak performance. Additionally, the model attained perfect recall (1.00) for Glioma and a micro-average AUC of 1.00. t-SNE visualizations and reliability diagrams confirm the model's ability to learn highly discriminative and well-calibrated features, positioning it as a trustworthy clinical decision support system.

Downloads

Download data is not yet available.

References

M. Tabassum, A. A. Suman, E. Suero Molina, E. Pan, A. Di Ieva, and S. Liu, “Radiomics and Machine Learning in Brain Tumors and Their Habitat: A Systematic Review,” Cancers, vol. 15, no. 15, p. 3845, Jul. 2023, doi: 10.3390/cancers15153845.

A. Mohebbi, S. Mohammadzadeh, A. H. Zare, Z. Moradi, A. A. Ardakani, and A. Mohammadi, “Assessing inter-rater reliability of MRI features in glioma: a multi-radiologist agreement study,” BMC Med Imaging, vol. 25, no. 1, p. 480, Nov. 2025, doi: 10.1186/s12880-025-01941-5.

D. N. Sindhura, R. M. Pai, S. N. Bhat, and M. M. M. Pai, “A review of deep learning and Generative Adversarial Networks applications in medical image analysis,” Multimedia Systems, vol. 30, no. 3, p. 161, Jun. 2024, doi: 10.1007/s00530-024-01349-1.

R. R. Ali et al., “Learning Architecture for Brain Tumor Classification Based on Deep Convolutional Neural Network: Classic and ResNet50,” Diagnostics, vol. 15, no. 5, p. 624, Mar. 2025, doi: 10.3390/diagnostics15050624.

M. B. Kurniawan and E. Utami, “Performance Comparison of ResNet50, VGG16, and MobileNetV2 for Brain Tumor Classification on MRI Images,” SISTEMASI, vol. 14, no. 2, p. 767, Mar. 2025, doi: 10.32520/stmsi.v14i2.5054.

N. Ullah, A. Javed, A. Alhazmi, S. M. Hasnain, A. Tahir, and R. Ashraf, “TumorDetNet: A unified deep learning model for brain tumor detection and classification,” PLoS ONE, vol. 18, no. 9, p. e0291200, Sep. 2023, doi: 10.1371/journal.pone.0291200.

A. Alshuhail et al., “Refining neural network algorithms for accurate brain tumor classification in MRI imagery,” BMC Med Imaging, vol. 24, no. 1, p. 118, May 2024, doi: 10.1186/s12880-024-01285-6.

C. K. K. Reddy et al., “A fine-tuned vision transformer based enhanced multi-class brain tumor classification using MRI scan imagery,” Front. Oncol., vol. 14, p. 1400341, Jul. 2024, doi: 10.3389/fonc.2024.1400341.

K. M. Hosny and M. A. Mohammed, “Explainable AI and vision transformers for detection and classification of brain tumor: a comprehensive survey,” Artif Intell Rev, vol. 58, no. 9, p. 259, Jun. 2025, doi: 10.1007/s10462-025-11221-x.

F. Ghazouani, P. Vera, and S. Ruan, “Efficient brain tumor segmentation using Swin transformer and enhanced local self-attention,” Int J CARS, vol. 19, no. 2, pp. 273–281, Oct. 2023, doi: 10.1007/s11548-023-03024-8.

X. Guo, X. Lin, X. Yang, L. Yu, K.-T. Cheng, and Z. Yan, “UCTNet: Uncertainty-guided CNN-Transformer hybrid networks for medical image segmentation,” Pattern Recognition, vol. 152, p. 110491, Aug. 2024, doi: 10.1016/j.patcog.2024.110491.

Y. Lyu and X. Tian, “MWG-UNet++: Hybrid Transformer U-Net Model for Brain Tumor Segmentation in MRI Scans,” Bioengineering, vol. 12, no. 2, p. 140, Jan. 2025, doi: 10.3390/bioengineering12020140.

A. F. Al Bataineh et al., “Enhanced Magnetic Resonance Imaging-Based Brain Tumor Classification with a Hybrid Swin Transformer and ResNet50V2 Model,” Applied Sciences, vol. 14, no. 22, p. 10154, Nov. 2024, doi: 10.3390/app142210154.

W. Ding et al., “FTransCNN: Fusing Transformer and a CNN based on fuzzy logic for uncertain medical image segmentation,” Information Fusion, vol. 99, p. 101880, Nov. 2023, doi: 10.1016/j.inffus.2023.101880.

Q. Pu, Z. Xi, S. Yin, Z. Zhao, and L. Zhao, “Advantages of transformer and its application for medical image segmentation: a survey,” BioMed Eng OnLine, vol. 23, no. 1, p. 14, Feb. 2024, doi: 10.1186/s12938-024-01212-4.

X. Liu, Y. Hu, and J. Chen, “Hybrid CNN-Transformer model for medical image segmentation with pyramid convolution and multi-layer perceptron,” Biomedical Signal Processing and Control, vol. 86, p. 105331, Sep. 2023, doi: 10.1016/j.bspc.2023.105331.

Ch. R. Prasad, K. Varshamrutha, B. Sindhuja, R. Ushasree, P. Nikhil, and A. Chakradhar, “MRI Based Brain Tumor classification Using a Fine-Tuned EfficientNetB3 Transfer Learning Model,” in 2024 5th International Conference for Emerging Technology (INCET), Belgaum, India: IEEE, May 2024, pp. 1–5. doi: 10.1109/INCET61516.2024.10592978.

S. Saeedi, S. Rezayi, H. Keshavarz, and S. R. Niakan Kalhori, “MRI-based brain tumor detection using convolutional deep learning methods and chosen machine learning techniques,” BMC Med Inform Decis Mak, vol. 23, no. 1, p. 16, Jan. 2023, doi: 10.1186/s12911-023-02114-6.

T. Sadad et al., “Brain tumor detection and multi‐classification using advanced deep learning techniques,” Microscopy Res & Technique, vol. 84, no. 6, pp. 1296–1308, Jun. 2021, doi: 10.1002/jemt.23688.

F. E. AlTahhan, G. A. Khouqeer, S. Saadi, A. Elgarayhi, and M. Sallah, “Refined Automatic Brain Tumor Classification Using Hybrid Convolutional Neural Networks for MRI Scans,” Diagnostics, vol. 13, no. 5, p. 864, Feb. 2023, doi: 10.3390/diagnostics13050864.

M. D. Irfani and U. Novia Wisesty, “Brain Tumor Classification Using EfficientNet-Based CNN Architecture in MRI Images,” in 2025 International Conference on Data Science and Its Applications (ICoDSA), Jakarta, Indonesia: IEEE, Jul. 2025, pp. 200–205. doi: 10.1109/ICoDSA67155.2025.11157267.

S. Takahashi et al., “Comparison of Vision Transformers and Convolutional Neural Networks in Medical Image Analysis: A Systematic Review,” J Med Syst, vol. 48, no. 1, p. 84, Sep. 2024, doi: 10.1007/s10916-024-02105-8.

A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” Jun. 03, 2021, arXiv: arXiv:2010.11929. doi: 10.48550/arXiv.2010.11929.

J. Li, J. Chen, Y. Tang, C. Wang, B. A. Landman, and S. K. Zhou, “Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives,” Medical Image Analysis, vol. 85, p. 102762, Apr. 2023, doi: 10.1016/j.media.2023.102762.

A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H. Roth, and D. Xu, “Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images,” Jan. 04, 2022, arXiv: arXiv:2201.01266. doi: 10.48550/arXiv.2201.01266.

T. Ren, V. Govindarajan, S. Bourouis, X. Wang, and S. Ke, “An interpretable hybrid deep learning framework for gastric cancer diagnosis using histopathological imaging,” Sci Rep, vol. 15, no. 1, p. 34204, Oct. 2025, doi: 10.1038/s41598-025-15702-5.

J. Bai, L. Yuan, S.-T. Xia, S. Yan, Z. Li, and W. Liu, “Improving Vision Transformers by Revisiting High-Frequency Components,” in Computer Vision – ECCV 2022, vol. 13684, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, Eds., in Lecture Notes in Computer Science, vol. 13684. , Cham: Springer Nature Switzerland, 2022, pp. 1–18. doi: 10.1007/978-3-031-20053-3_1.

Q. Jia and H. Shu, “BiTr-Unet: A CNN-Transformer Combined Network for MRI Brain Tumor Segmentation,” in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, vol. 12963, A. Crimi and S. Bakas, Eds., in Lecture Notes in Computer Science, vol. 12963. , Cham: Springer International Publishing, 2022, pp. 3–14. doi: 10.1007/978-3-031-09002-8_1.

A. Kurz et al., “Uncertainty Estimation in Medical Image Classification: Systematic Review,” JMIR Med Inform, vol. 10, no. 8, p. e36427, Aug. 2022, doi: 10.2196/36427.

S. Wang, Z. Dou, J. Liu, Q. Zhu, and J.-R. Wen, “Personalized and Diversified: Ranking Search Results in an Integrated Way,” ACM Trans. Inf. Syst., vol. 42, no. 3, pp. 1–25, May 2024, doi: 10.1145/3631989.

H. Fang, D. Becker, S. Wermter, and T. Gerkmann, “Integrating Uncertainty into Neural Network-based Speech Enhancement,” IEEE/ACM Trans. Audio Speech Lang. Process., vol. 31, pp. 1587–1600, 2023, doi: 10.1109/TASLP.2023.3265202.

T. Dolar, J. Chen, and W. Chen, “Uncertainty quantification driven machine learning for improving model accuracy in imbalanced regression tasks,” Expert Systems with Applications, vol. 261, p. 125526, Feb. 2025, doi: 10.1016/j.eswa.2024.125526.

E. M. G. Younis, M. N. Mahmoud, A. M. Albarrak, and I. A. Ibrahim, “A Hybrid Deep Learning Model with Data Augmentation to Improve Tumor Classification Using MRI Images,” Diagnostics, vol. 14, no. 23, p. 2710, Nov. 2024, doi: 10.3390/diagnostics14232710.

Y. Liu, Y.-H. Wu, G. Sun, L. Zhang, A. Chhatkuli, and L. Van Gool, “Vision Transformers with Hierarchical Attention,” Mach. Intell. Res., vol. 21, no. 4, pp. 670–683, Aug. 2024, doi: 10.1007/s11633-024-1393-8.

P. Priyadarshini, P. Kanungo, and T. Kar, “Multigrade brain tumor classification in MRI images using Fine tuned efficientnet,” e-Prime - Advances in Electrical Engineering, Electronics and Energy, vol. 8, p. 100498, Jun. 2024, doi: 10.1016/j.prime.2024.100498.

A. A. Waskita, J. M. Amda, D. S. K. Sihono, and H. Prasetio, “EfficientNetV2 based for MRI brain tumor image classification,” in 2023 International Conference on Computer, Control, Informatics and its Applications (IC3INA), Bandung, Indonesia: IEEE, Oct. 2023, pp. 171–176. doi: 10.1109/IC3INA60834.2023.10285782.

A. K. Sharma and N. K. Verma, “A Novel Vision Transformer with Residual in Self-attention for Biomedical Image Classification,” Pattern Recognition, vol. 172, p. 112497, Apr. 2025, doi: 10.1016/j.patcog.2025.112497.

A. S. M. S. Sagar, M. Z. Islam, J. Tanveer, and H. S. Kim, “Uncertainty-Aware Adaptive Multiscale U-Net for Low-Contrast Cardiac Image Segmentation,” Applied Sciences, vol. 15, no. 4, p. 2222, Feb. 2025, doi: 10.3390/app15042222.

A. Nur, K. Nurhanafi, and E. R. Putri, “Brain Tumor Segmentation in MR Images Using Swin Transformer,” Atom Indo., vol. 51, no. 2, pp. 97–108, May 2025, doi: 10.55981/aij.2025.1580.

Y. Huang et al., “Comparative Analysis of ImageNet Pre-Trained Deep Learning Models and DINOv2 in Medical Imaging Classification,” Feb. 13, 2024, arXiv: arXiv:2402.07595. doi: 10.48550/arXiv.2402.07595.

M. M. Zahoor et al., “Brain Tumor MRI Classification Using a Novel Deep Residual and Regional CNN,” Biomedicines, vol. 12, no. 7, p. 1395, Jun. 2024, doi: 10.3390/biomedicines12071395.

A. S. Sambyal, U. Niyaz, N. C. Krishnan, and D. R. Bathula, “Understanding calibration of deep neural networks for medical image classification,” Computer Methods and Programs in Biomedicine, vol. 242, p. 107816, Dec. 2023, doi: 10.1016/j.cmpb.2023.107816.

L. Yang, T. Wang, J. Zhang, S. Kang, S. Xu, and K. Wang, “Deep learning–based automatic segmentation of meningioma from T1-weighted contrast-enhanced MRI for preoperative meningioma differentiation using radiomic features,” BMC Med Imaging, vol. 24, no. 1, p. 56, Mar. 2024, doi: 10.1186/s12880-024-01218-3.

M. R. Shoaib et al., “Improving brain tumor classification: An approach integrating pre-trained CNN models and machine learning algorithms,” Heliyon, vol. 11, no. 10, p. e33471, May 2025, doi: 10.1016/j.heliyon.2024.e33471.

P. S. Tejashwini, J. Thriveni, and K. R. Venugopal, “EBT Deep Net: Ensemble brain tumor Deep Net for multi-classification of brain tumor in MR images,” Biomedical Signal Processing and Control, vol. 95, p. 106312, Sep. 2024, doi: 10.1016/j.bspc.2024.106312.

M. A. Gómez-Guzmán et al., “Enhanced Multi-Class Brain Tumor Classification in MRI Using Pre-Trained CNNs and Transformer Architectures,” Technologies, vol. 13, no. 9, p. 379, Aug. 2025, doi: 10.3390/technologies13090379.

M. Nickparvar, "Brain Tumor MRI Dataset," Kaggle, 2021. [Online]. Available: https://www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset. [Accessed: Dec. 03, 2025].