融合跨模态Transformer的外部知识型VQA

王虞 ,  李明锋 ,  孙海春

科学技术与工程 ›› 2024, Vol. 24 ›› Issue (20) :8577-8586

PDF (3137KB)
科学技术与工程 ›› 2024, Vol. 24 ›› Issue (20) :8577-8586 DOI: 10.12404/j.issn.1671-1815.2305443
自动化技术、计算机技术

融合跨模态Transformer的外部知识型VQA

作者信息 +

External Knowledge-based VQA Integrating Cross Modal Transformers

Author information +
文章历史 +

摘要

针对外部知识型的视觉问答(visual question answering, VQA)任务性能效果不佳的问题,构建一种融合跨模态Transformer的外部知识型VQA模型框架,通过在VQA模型外引入外接知识库来提高VQA模型在外部知识型任务上的推理能力。进一步地,模型借助双向交叉注意力机制提升文本问题、图像、外接知识的语义交互融合能力,用于优化VQA模型在面对外部知识时普遍存在的推理能力不足的问题。结果表明:与基线模型LXMERT相比,在OK VQA数据集上,本文模型整体性能指标overall提升了15.01%。同时,与已有最新模型相比,在OK VQA数据集上,本文模型整体性能指标overall提升了4.46%。可见本文模型在改进外部知识型VQA任务性能方面有所提升。

Abstract

In response to the issue of poor performance of external knowledge-based visual question answering tasks, a framework is constructed for external knowledge-based visual question answering (VQA) models that integrated cross-modal Transformers. By introducing an external knowledge base outside the VQA model, the inference ability of the VQA model on external knowledge-based tasks was improved. Further, the model utilized a bidirectional cross attention mechanism to enhance the semantic interactive and fusion ability of text problems, images, and in order to optimize the problem of insufficient reasoning ability commonly found in VQA models in the face of external knowledge. The results show that compared with the baseline model LXMERT, the overall performance index of the proposed model overall improves by 15.01% on the OK VQA dataset. Meanwhile, compared with the existing latest model, the overall performance index of the proposed model overall improves by 4.46% on the OK VQA dataset. It can be seen that the proposed model improves the performance of external knowledge-based VQA tasks.

引用本文

引用格式 ▾
王虞, 李明锋, 孙海春. 融合跨模态Transformer的外部知识型VQA[J]. 科学技术与工程, 2024, 24(20): 8577-8586 DOI:10.12404/j.issn.1671-1815.2305443
WANG Yu, LI Ming-feng, SUN Hai-chun. External Knowledge-based VQA Integrating Cross Modal Transformers[J]. Science Technology and Engineering, 2024, 24(20): 8577-8586 DOI:10.12404/j.issn.1671-1815.2305443

登录浏览全文

4963

注册一个新账户 忘记密码

参考文献

基金资助

公安部技术研究计划项目(2020JSYJC22)

中央高校基本科研业务费专项资金(2022JKF02015)

PDF (3137KB)

30

访问

0

被引

详细

导航
相关文章

/

〈 〉