基于轻量化改进ERNIE-RCNN的中文新闻标题分类研究
DOI:
作者:
作者单位:

华北电力大学 控制与计算机工程学院

作者简介:

通讯作者:

中图分类号:

TP391

基金项目:


Research on Chinese News Title Classification Based on Lightweight
Author:
Affiliation:

North China Electric Power University School of Control and Computer Engineering

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    针对大型预训练语言模型在处理新闻标题时,面临参数规模庞大、无法高效利用上下文语意特征以及循环卷积神经网络对初始输入元素重要性忽视的问题,本文提出了一种融合混合专家模型(MoE)的ERNIE与注意力机制的循环卷积神经网络(RCNN)的新闻标题分类方法。首先通过融合MoE的ERNIE进行文本编码,随后利用注意力RCNN在保留文本词序和特征的基础上进行分类。为提高分类能力,通过计算输入的融合上下文权重对RCNN进行改进。在计算MoE中各个专家权重的过程中,采用Gumbel_Softmax作为门控函数替代普通的Softmax函数,从而更好地控制平滑程度。实验结果表明,相较于传统分类方法,本研究提出的分类方法展现出显著优势,极大地减少了参数数量。在此基础上,F1值相较于传统模型提升了0.51%。最后,通过消融实验证实了该分类方法在分类任务上的可行性。

    Abstract:

    Aiming at the problems of large pre-trained language models in processing news headlines, such as the large scale of parameters, the inability to efficiently utilize contextual semantic features, and the neglect of the importance of initial input elements by cyclic convolutional neural networks, In this study, we propose a novel headline classification method that integrates the hybrid expert model (MoE) ERNIE with the attention mechanism of the cyclic Convolutional Neural network (RCNN). In this study, text encoding is performed by ERNIE with MoE, and then text classification is performed by using attentional RCNN on the basis of retaining word order and features. To improve the classification capability, we improve the RCNN by calculating the input fusion context weight. In the process of calculating the weight of each expert in MoE, this study uses Gumbel_Softmax as a gating function to replace the ordinary Softmax function, so as to better control the degree of smoothness. The experimental results show that compared with traditional classification methods, the classification method proposed in this study shows significant advantages, greatly reducing the number of parameters. On this basis, F1 value increased by 0.51% compared with the traditional model. Finally, through the ablation experiment, the feasibility of this classification method in the classification task is confirmed.

    参考文献
    相似文献
    引证文献
引用本文

张之欣,李莉,王小龙. 基于轻量化改进ERNIE-RCNN的中文新闻标题分类研究[J]. 科学技术与工程, , ():

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2023-10-07
  • 最后修改日期:2024-05-29
  • 录用日期:2024-06-05
  • 在线发布日期:
  • 出版日期:
×
亟待确认版面费归属稿件,敬请作者关注