
科学技术与工程

中国技术经济学会
基于多特征融合的脑电图高质量视频重建技术
High Quality EEG Video Reconstruction Technique Based on Multi-Feature Fusion
为探索大脑与视觉之间的联系,提高大脑活动重建视频的清晰度与准确性,提出了一种名为高质量脑电视频重建(high quality electroencephalogram video reconstruction, HQEEGVR)的方法进行脑电信号重建视频。首先,提出三分支脑电特征提取网络——掩蔽时空频融合网络(masking spatio-temporal frequency fusion network, MSTFFNet)从脑电信号中提取大脑活动信息,深入挖掘大脑活动变化背后的语义,同时提取时空频信息;其次,引入跨模态对比学习,对齐脑电、文本、图像特征,以便生成阶段使用;然后,提出级联视频扩散模型,具体来说,先利用稳定扩散模型以脑电特征为条件生成参考视频帧,接着以视频帧为参考,融入运动矢量,引入视频扩散模型捕捉视频时间特征;最终生成高质量视频。结果表明,该模型在重建视频的主体、动作、颜色、语义等方面表现较好。可见利用脑电信号可以捕获大脑活动的视觉与语义信息,从而重建高保真度和视觉真实性的视频。
In order to explore the connection between brain and vision and improve the clarity and accuracy of brain activity reconstruction video, a new method called high quality electroencephalogram video reconstruction (HQEEGVR) was proposed to reconstruct video from EEG (electroencephalogram) signals. Firstly, the masking spatio-temporal frequency fusion network (MSTFFNet), a three-branch EEG feature extraction network, was proposed to extract brain activity information from EEG signals and dig deeper into the semantics behind brain activity changes, spatio-temporal frequency information was extracted at the same time. Secondly, cross-modal contrast learning was introduced to align EEG, text and image features for use in the generation stage. Then, a cascade video diffusion model was proposed, specifically, the stable diffusion model was used to generate reference video frames based on EEG features, and then the video frames were used as references, motion vectors were integrated, and the video diffusion model was introduced to capture the video time features. High quality videos were ultimately generated. The results show that the model performs well in the reconstruction of the subject, motion, color and semantics of the video. It can be seen that the EEG signal can be used to capture the visual and semantic information of the brain activity, so as to reconstruct the video with high fidelity and visual authenticity.
脑电信号 / 掩蔽时空频融合网络 / 稳定扩散模型 / 视频扩散模型 / 运动矢量 / 视频重建
EEG / MSTFFNet / stable diffusion model / video diffusion model / motion vector / video reconstruction
/
| 〈 |
|
〉 |