Abstract:Compared to image instance segmentation in general scenes, instance segmentation in complex stacked scenes is affected by complex situations such as severe occlusion and stacking of similar objects, making instance segmentation more difficult. This article proposes an instance segmentation algorithm that combines YOLOv8 with a dual layer feature network strategy for garbage instance segmentation in complex stacking scenarios. Firstly, in the data preprocessing section, feature data is layered and dual branch feature fusion is achieved through a dual layer Graph Convolutions Network (GCN) to reduce the impact of stacking on the features of occluded objects, thus solving the problem of practical segmentation under complex stacking occlusion. At the same time, in order to solve the problem of confusion among similar test objects, soft thresholding non maximum suppression algorithms and new intersection and union ratio algorithms have been incorporated. Finally, based on the complexity of the application scenario and dataset, the feature extraction module of the backbone network was optimized, and a multiscale attention mechanism was introduced in the backbone network, effectively improving the detection performance of the model. This experiment used occluded garbage classification instances to segment the dataset, and the experimental results showed that the average accuracy, average precision 50 (AP50), and AP50-95 of this method were better than other previous methods when the intersection to union ratio threshold was 0.5. Compared to the original YOLOv8 algorithm, the detection AP50 has increased by 7.9% and the segmentation AP50 has increased by 5.4%, resulting in better detection and segmentation performance.