Theoretical analysis was conducted on the failure mechanism of rock mass anchoring system under plane strain conditions. By integrating the generalized nonlinear unified strength theory with the upper bound theorem of limit analysis and variational method principles, a novel theoretical framework was developed for the first time, incorporating an optimized objective function with velocity discontinuity equations and a kinematically admissible velocity field for anchor failure. Based on this theoretical framework, the determination method of the ultimate pullout resistance of rock bolts under synergistic failure of the anchorage body and surrounding rock mass was systematically investigated, while quantitatively revealing the influence patterns of key material parameters on load-bearing performance. Numerical results demonstrate that the ultimate pullout resistance shows positive correlations with rock mass parameters (k, s, m), uniaxial compressive strength, and unit weight, but exhibits negative correlation with parameter a. Notably, increasing the anchorage length within a critical range significantly enhances the pullout capacity. The analytical model established provides theoretical guidance for deep rock mass anchorage engineering design, and the identified parameter sensitivity patterns offer valuable insights for optimizing bolt support parameters in practical engineering applications.