Audio-based High-resolution Face Portrait Method

Hao-ming QIN , Fan-liang BU , Fang-hao ZHONG

Science Technology and Engineering ›› 2025, Vol. 25 ›› Issue (13) : 5515 -5526.

PDF
Science Technology and Engineering ›› 2025, Vol. 25 ›› Issue (13) : 5515-5526 DOI: 10.12404/j.issn.1671-1815.2403342
research-article
Audio-based High-resolution Face Portrait Method
Author information +
History +
PDF

Abstract

Existing voice-driven facial generation methods still face challenges in feature extraction and generation quality, and have yet to fully explore the deep correlation between audio and facial features. To address above mentioned issues, a research approach that combines Mel frequency cepstral coefficients (MFCC) was proposedfor audio feature extraction with the image generation capabilities of the second generation of style generative adversarial networks (StyleGAN2) was proposed. In terms of audio processing, MFCC was employed as the feature extraction method. To more effectively extract and transmit features from the audio, a ResNet18-based residual module was designed and integrated with the squeeze-and-excitation (SE) attention mechanism. Additionally, the activation function in the original residual blocks was optimized and improved by using the Mish activation function, aiming to mitigate the gradient vanishing problem in deep networks, maintain the integrity of feature information, and enhance the accuracy and generalization ability of the model. The StyleGAN2 model was then utilized as the facial image generation model. Experimental results demonstrate that the integration of the designed audio processing network with the StyleGAN2 facial generation model exhibits outstanding performance in the task of voice-driven facial generation. Through comprehensive evaluation using metrics such as Fréchet inception distance (FID) and path length, the proposed method shows a significant improvement in generation quality compared to existing methods, thus fully proving its effectiveness and superiority.

Keywords

voice-to-face generation / Mel frequency cepstral coefficients / style generative adversarial networks / attention mechanism

Cite this article

Download citation ▾
Hao-ming QIN, Fan-liang BU, Fang-hao ZHONG. Audio-based High-resolution Face Portrait Method. Science Technology and Engineering, 2025, 25(13): 5515-5526 DOI:10.12404/j.issn.1671-1815.2403342

登录浏览全文

4963

注册一个新账户 忘记密码

References

PDF (9016KB)

19

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉