Aiming at the problems of large pre-trained language models in processing news headlines, such as the large scale of parameters, the inability to efficiently utilize contextual semantic features, and the neglect of the importance of initial input elements by cyclic convolutional neural networks, In this study, we propose a novel headline classification method that integrates the hybrid expert model (MoE) ERNIE with the attention mechanism of the cyclic Convolutional Neural network (RCNN). In this study, text encoding is performed by ERNIE with MoE, and then text classification is performed by using attentional RCNN on the basis of retaining word order and features. To improve the classification capability, we improve the RCNN by calculating the input fusion context weight. In the process of calculating the weight of each expert in MoE, this study uses Gumbel_Softmax as a gating function to replace the ordinary Softmax function, so as to better control the degree of smoothness. The experimental results show that compared with traditional classification methods, the classification method proposed in this study shows significant advantages, greatly reducing the number of parameters. On this basis, F1 value increased by 0.51% compared with the traditional model. Finally, through the ablation experiment, the feasibility of this classification method in the classification task is confirmed.