Systematic Literature Review of the Comparison Between Yolov11 and Mask R-Cnn for Image Segmentation

Abstract

Image segmentation is a fundamental task in computer vision that demands a balance between accuracy and computational efficiency. YOLOv11 as a single-stage detector and Mask R-CNN as a two-stage approach represent two distinct paradigms, yet a comprehensive comparison of the two in the context of segmentation has rarely been synthesized systematically. This study aims to conduct a systematic literature review of the comparative performance of YOLOv11 and Mask R-CNN for image segmentation. The method follows the PRISMA protocol, searching the Scopus, IEEE Xplore, ScienceDirect, and Google Scholar databases over the 2023–2026 period; of 312 initial records, 28 studies met the inclusion criteria and were analyzed. The synthesis shows that YOLOv11 consistently excels in mean Average Precision (mAP), inference speed, and parameter efficiency, whereas Mask R-CNN tends to produce smoother mask contours on single objects but suffers from over-segmentation and a higher computational load. In one cited study, YOLOv11 achieved an mAP50 of 71.6% compared with 41.1% for Mask R-CNN, with a training time roughly one third shorter. These findings provide context-based guidance for model selection, particularly for real-time systems on resource-constrained devices.