In view of the fact that the density of high-density particles is large, the number is large, the shape is different, and the color is similar, the traditional method for the segmentation of sandstone particles has the disadvantages of inaccurate detection and missing detection. To obtain better segmentation results with limited samples, it becomes more challenging. Based on the above problems, a DGC-Mask R-CNN detection model based on improved Mask R-CNN was proposed, aiming at the segmentation and recognition of a small number of samples and high-density sandstone particles. 128 ultra-high resolution images were collected, each with nearly 200 sandstone particle instances, for a total of 26 200 instances, and Albu was used for image augmentation to improve the model's generalization ability and prevent overfitting with limited samples. The Barlow Twins self-supervised pre-trained model was used to extract the sandstone particle features. In Mask R-CNN, a ResNet50 model was built as the backbone feature extraction network, and the traditional convolution method was improved by using the deformable convolutional neural network (DCN) and adding the global context block (GCB) attention mechanism in the C₃, C₄, and C₅ feature convolution layers of ResNet50's BottleNeck. The improved upsampling algorithm CARAFE was used in multiple cascaded upsampling modules of the upsampler. The experimental results show that the improved DGC-Mask R-CNN achieves an average precision of 88.9% and 88.8% for detection and segmentation identification, respectively, which is higher than traditional Mask R-CNN, Cascade-Mask R-CNN, Mask Scoring R-CNN, and HybridTaskCascade. Compared with other models, the average accuracy of the model is significantly improved. The results obtained from the segmented model can be used to statistically analyze the sandstone particles and calculate the length and width axis to trace the source of the sandstone particles and evaluate the distance of sandstone migration caused by crustal movement in underground oil reservoirs.