
科学技术与工程

中国技术经济学会
基于开源大模型的110警情关键要素抽取方法
Extracting Key Elements of 110 Police Incidents Based on Open Source Large Language Models
随着互联网技术的迅猛发展和警情数量的持续增加,依赖大量人力和传统自然语言处理方法来提取警情要素已不再适用,利用大模型等先进技术提升警情要素抽取效果已成为警务数据分析领域发展的必然趋势。针对警情的特点及警情要素的抽取需求,采用了构建公安领域大模型的方法来进行警情要素抽取。在基座模型方面采用了多种轻量级(6/7b)开源大模型进行测试,在提升模型效果方面以LoRA微调为基础,结合数据工程提出了针对警情的基于ChatGPT的零样本数据增强方法和提示词工程方法来提升模型效果,以2019年北京市重点警情数据作为样例进行了测试。相对基础模型,提示词工程可提升F1 3% ~ 30%,微调可提升F1 10% ~ 50%,对不同基座模型进行微调后比对,Baichuan模型在警情要素抽取方面的综合表现最好;使用数据增强方法在扩充2倍数据时进一步提升了4%的F1,同时模型能力达到最优。相对微调后的通用信息抽取(universal information extraction, UIE)模型,所构建的警情要素抽取大模型对各个要素抽取均提升了5%左右的F1,对于警情“组织”要素效果提升了42%的F1。
With the rapid development of Internet technology and the continuous increase in the number of police cases, relying on a large amount of manpower and traditional natural language processing methods to extract police case elements is no longer applicable. Utilizing advanced technologies such as large language models to improve the extraction effect of police case elements has become an inevitable trend in the development of the field of police data analysis. In view of the characteristics of police cases and the extraction requirements of police case elements, the method of constructing a large language model in the public security field was adopted to extract police case elements. For the base model, a variety of lightweight (6/7b) open-source large language models were tested. To improve the model performance, based on LoRA fine-tuning, combined with data engineering, a zero-shot data augmentation method and a prompt engineering method based on ChatGPT for police cases were proposed to enhance the model performance. The key police case data of Beijing in 2019 were used as examples for testing. Compared with the base model, prompt engineering can increase the F1 by 3% ~ 30%, and fine-tuning can increase the F1 by 10% ~ 50%. After fine-tuning different base models and making comparisons, the Baichuan model shows the best comprehensive performance in the extraction of police case elements. When using the data augmentation method to expand the data by 2 times, the F1 is further increased by 4%, and the model ability reaches the optimal level. Compared with the fine-tuned general information extraction large language of universal information extraction model(UIE), the large language model for police case element extraction constructed increases the F1 of each element extraction by about 5%, and for the "organization" element of police cases, the F1 is increased by 42%.
large language models / police incident / information extraction
/
| 〈 |
|
〉 |