Several limitations of commonly used modalities, including visual cues, physiological signals, and vehicle behavior, are encountered in real-world driver fatigue detection scenarios. Valuable information related to the driver's fatigue state is also contained in speech signals. To improve the accuracy and robustness of fatigue detection methods, a lightweight multimodal approach integrating audio and visual modalities was proposed. This method was designed to leverage the complementary strengths of acoustic and ocular features for more precise and reliable driver fatigue detection. For the audio modality, the CNN-SE-MN4 model was proposed by incorporating the efficient and lightweight MobileNet v4 architecture into a mainstream CNN-SE network, enabling fatigue assessment based on spectrogram analysis. For the visual modality, the YOLOv8n-MN4 model was developed by replacing the backbone of YOLOv8n with MobileNet v4 and introducing a parameter-free attention mechanism into the neck network, facilitating the detection of the driver's eye region and subsequent fatigue state inference from eye images. Finally, weights were assigned to fatigue indicators from both modalities through fuzzy analytic hierarchy process, enabling comprehensive fatigue level evaluation. The results show that the improvements made to mainstream models using MobileNet v4 are all effective. Compared with the traditional equal-weight fusion method, the F₁-score is improved by 5.7 percentage points, confirming the effectiveness of incorporating the audio modality.