To address the issue of insufficient pathological speech feature extraction in existing research, which leads to low recognition rates for pathological speech, a pathological speech recognition algorithm based on a dual-branch residual network was proposed. Considering the complex and diverse speech symptoms of dysarthria patients, wideband and narrowband spectrograms were used as inputs to the network. An adaptive feature extraction residual block was introduced, which comprehensively captures pathological features from multiple dimensions, including position, channel, filter, and pixel, through full-dimensional dynamic pixel attention convolution. A dual-stream complementary fusion module was proposed, where the weighted fusion of features not only preserves critical information from each branch but also achieves optimal feature representation through cross-dimensional interactions, thereby improving the accuracy of pathological speech recognition. Experiments conducted on the Chinese pathological speech dataset THE-POSSD and the publicly available Western pathological speech dataset UA-Speech demonstrate the effectiveness and generalization capability of the proposed algorithm.