基于单流网络的语音-人脸的跨模态学习方法

钟方昊 ,  卜凡亮 ,  秦昊铭

科学技术与工程 ›› 2025, Vol. 25 ›› Issue (11) :4638-4646

PDF (5730KB)
科学技术与工程 ›› 2025, Vol. 25 ›› Issue (11) :4638-4646 DOI: 10.12404/j.issn.1671-1815.2403487
自动化技术、计算机技术

基于单流网络的语音-人脸的跨模态学习方法

作者信息 +

Cross Modal Learning Method of Speech Face via Single Stream Network

Author information +
文章历史 +

摘要

现有的语音-人脸跨模态关联学习方法多采用双流网络结构,在降低计算复杂度、模型轻量化和高效特征融合方面还面临一些挑战,为了改善模型性能,提高跨模态学习的效率,提出一种基于单流网络的语音-人脸的跨模态学习方法。首先,将预处理的两种模态数据送入单流特征提取网络,利用基于类信息的损失函数学习提取两种模态的有效特征,接着对提取的两种模态特征向量进行基于注意力机制的特征融合,最后使用余弦相似度算法和交叉熵损失相结合的方法来学习两种模态的关联,从而完成跨模态关联学习任务。实验结果表明,本文提出的方法在语音-人脸跨模态验证、匹配和检索任务上均取得了良好的效果,在考虑网络结构轻量化和灵活性的同时保证了优秀的性能。

Abstract

Existing methods for audio-visual cross-modal association learning often adopt a dual-stream network structure, but they still face challenges in reducing computational complexity, model light weighting, and efficient feature fusion. To improve model performance and enhance the efficiency of cross-modal learning, a single-stream network-based approach for audio-visual cross-modal learning was proposed. Firstly, preprocessed data from both modalities were fed into a single-stream feature extraction network, where a class-information-based loss function was employed to learn and extract feature vectors from both modalities. Subsequently, attention-based feature fusion was performed on the extracted feature vectors from both modalities. Finally, a combination of cosine similarity algorithm and cross-entropy loss was used to learn the association between the two modalities, thus completing the cross-modal association learning task. Experimental results demonstrate that the proposed method achieves promising performance in audio-visual cross-modal verification, matching, and retrieval tasks, ensuring excellent performance while considering the lightness and flexibility of the network structure.

关键词

关联学习 / 语音-人脸跨模态 / 单流网络 / 特征融合

Key words

association learning / voice-face cross-modal / single-stream network / feature fusion

引用本文

引用格式 ▾
钟方昊, 卜凡亮, 秦昊铭. 基于单流网络的语音-人脸的跨模态学习方法[J]. 科学技术与工程, 2025, 25(11): 4638-4646 DOI:10.12404/j.issn.1671-1815.2403487
ZHONG Fang-hao, BU Fan-liang, QIN Hao-ming. Cross Modal Learning Method of Speech Face via Single Stream Network[J]. Science Technology and Engineering, 2025, 25(11): 4638-4646 DOI:10.12404/j.issn.1671-1815.2403487

登录浏览全文

4963

注册一个新账户 忘记密码

参考文献

基金资助

中国人民公安大学安全防范工程双一流创新研究专项(2023SYL08)

PDF (5730KB)

36

访问

0

被引

详细

导航
相关文章

/

〈 〉