
科学技术与工程

中国技术经济学会
基于音频的高分辨率人脸画像方法
Audio-based High-resolution Face Portrait Method
现有的语音驱动人脸生成方法在特征提取与生成质量上仍面临挑战,且尚未充分挖掘音频与人脸特征之间的深层关联。为解决这些问题,提出一种结合梅尔频率倒谱系数(Mel frequency cepstral coefficients, MFCC)音频特征提取与第二代样式生成对抗网络(style generative adversarial net-works 2, StyleGAN2)图像生成技术的研究方法。在音频处理方面,采用了梅尔频率倒谱系数作为特征提取方法。为了更有效地从音频中提取和传递特征,设计了一种基于ResNet18的残差模块,并融入了SE(squeeze-and-excitation)注意力机制。同时对原残差块中的激活函数进行了优化改进,采用Mish激活函数,旨在减少深层网络中的梯度消失问题,保持特征信息的完整性并提高模型的准确性和泛化能力。采取StyleGAN2模型作为人脸图像的生成模型。实验结果表明,结合了设计的音频处理网络和StyleGAN2的人脸生成模型,在语音驱动的人脸生成任务中展现出了卓越的性能。通过综合评估Fréchet起始距离(Fréchet inception distance, FID)和路径长度等指标,本文方法在语音驱动的人脸生成任务中相较于现有方法,在生成质量上有显著提升,充分证明了所提方法的有效性和优越性。
Existing voice-driven facial generation methods still face challenges in feature extraction and generation quality, and have yet to fully explore the deep correlation between audio and facial features. To address above mentioned issues, a research approach that combines Mel frequency cepstral coefficients (MFCC) was proposedfor audio feature extraction with the image generation capabilities of the second generation of style generative adversarial networks (StyleGAN2) was proposed. In terms of audio processing, MFCC was employed as the feature extraction method. To more effectively extract and transmit features from the audio, a ResNet18-based residual module was designed and integrated with the squeeze-and-excitation (SE) attention mechanism. Additionally, the activation function in the original residual blocks was optimized and improved by using the Mish activation function, aiming to mitigate the gradient vanishing problem in deep networks, maintain the integrity of feature information, and enhance the accuracy and generalization ability of the model. The StyleGAN2 model was then utilized as the facial image generation model. Experimental results demonstrate that the integration of the designed audio processing network with the StyleGAN2 facial generation model exhibits outstanding performance in the task of voice-driven facial generation. Through comprehensive evaluation using metrics such as Fréchet inception distance (FID) and path length, the proposed method shows a significant improvement in generation quality compared to existing methods, thus fully proving its effectiveness and superiority.
语音生成人脸 / 梅尔频率倒谱系数 / 样式生成对抗网络 / 注意力机制
voice-to-face generation / Mel frequency cepstral coefficients / style generative adversarial networks / attention mechanism
/
| 〈 |
|
〉 |