Brain fatigue in visual display terminal (VDT) operators causes a sharp decline in cognitive efficiency and threatens human-machine system safety, making fatigue detection critically significant. Existing physiological signal-based methods are limited by low effectiveness and high inter-subject feature distribution variability. To address these issues, feature selection and fusion were first performed. Classification performance was improved using ensemble learning, and inter-subject variability was reduced through feature transfer methods. A brain fatigue detection experiment based on multimodal physiological signals was conducted. Fatigue was induced by designing a continuous computational task combining mental arithmetic and 2-Back tests. Physiological data—including electrocardiogram (ECG), electroencephalogram (EEG), galvanic sskin response (GSR), respiration (RESP), and electromyography (EMG)—were collected from 11 volunteers. When only ensemble learning was applied, the fatigue detection model achieved 86.32% accuracy. After integrating feature transfer, the maximum mean discrepancy (MMD) between cross-subject features was reduced by 63.1%, and accuracy improved to 90.60%. The results confirm that the proposed method achieves higher recognition accuracy, providing a robust framework for brain fatigue detection in VDT operators using multimodal physiological signals.