相较于一般场景下的图像实例分割，复杂堆叠场景下的实例分割受到严重遮挡、同类别待测物体堆叠等复杂情况的影响，使得其实例分割具有更大的难度。针对具有复杂堆叠场景下的垃圾实例分割问题，提出了一种融合YOLOv8与双层特征网络策略的实例分割算法。首先，在数据预处理部分进行特征数据分层，并通过双层图卷积网络（graph convolutions network, GCN）实现双分支特征融合，减弱堆叠情况对被遮挡物体特征的影响，从而解决复杂堆叠遮挡下的实例分割问题。同时，为了解决同类待测物体易混淆的问题，融入了软阈值化非极大值抑制算法和新的交并比算法。最后，根据应用场景和数据集的复杂性，优化了主干网络部分的特征提取模块，并在主干网络部分引入了多尺度注意力机制，有效提高了模型的检测性能。实验使用遮挡垃圾分类实例分割数据集，实验结果表明该方法的平均准确率、交并比阈值为0.5时的平均准确率（AP₅₀）、交并比为0.5 ~ 0.95时的平均准确率（AP_50~95）等指标较之前的其他方法更优。相较于原YOLOv8算法，检测AP₅₀提高了7.9%，分割AP₅₀提高了5.4%，具有更好的检测和分割效果。

Compared to image instance segmentation in general scenes, instance segmentation in complex stacked scenes is affected by complex situations such as severe occlusion and stacking of similar objects, making instance segmentation more difficult. To solve the problem of garbage instance segmentation in complex stacking scenarios, an instance segmentation algorithm combining YOLOv8 and two-layer feature network strategy was proposed. Firstly, the feature data was layered in the data preprocessing part, and the two-branch feature fusion was realized through the graph convolutions network (GCN), which reduces the influence of stacking on the features of the occluded objects, thus solving the instance segmentation problem under complex stack occlusion. At the same time, in order to solve the problem that similar objects are easily confused, a soft threshold non-maximum suppression algorithm and a new intersection ratio algorithm were integrated. Finally, according to the complexity of application scenarios and data sets, the feature extraction module of the backbone network was optimized, and the multi-scale attention mechanism was introduced in the backbone network, which effectively improves the detection performance of the model. In the experiment, examples of occlusive garbage classification were used to segment the dataset. The experimental results show that this method outperforms other methods in terms of average accuracy, average accuracy when the intersection to union ratio threshold is 0.5 (AP₅₀), and average accuracy when the intersection to union ratio is 0.5 ~ 0.95 (AP_50~95). Compared with the original YOLOv8 algorithm, the detection AP₅₀ is increased by 7.9% and the segmentation AP₅₀ is increased by 5.4%, which has better detection and segmentation effects.