To address the problem of underutilization of features in traditional micro-image recognition algorithms for Chinese herbal medicines due to the neglect of text data, a multimodal adaptive fusion method for micro-image recognition of Chinese herbal medicines was proposed. The problem of low recognition accuracy due to the large intra-class differences and small inter-class differences in the microscopic feature images of Chinese herbal medicines was addressed by collaboratively analyzing the image and text data. Firstly, a semantic-driven attention-based dynamic mask generator was introduced, which generated dynamic masks from text input and served as an effective constraint to mitigate misclassification between similar microscopic images in the final classification. Secondly, a microscopic image feature encoder was designed to extract both overall information and fine-grained features of microscopic images. Lastly, a self-attention mechanism was incorporated into the multimodal fusion module, where a text-driven network was utilized to filter image features, thereby reducing redundant information and enhancing classification accuracy. Experimental results show that recognition accuracy of 97.5% is achieved by the model, showing improvements of 7.7, 6.1, 4.3, and 2.9 percentage points compared to the Biformer-B, InceptionNeXt-B, EfficientFormer V2-L, and ConvNeXt V2-L models, respectively. In summary, the proposed multimodal adaptive fusion method effectively compensates for the feature defects of single-image input and provides a reliable technical solution for the high-precision recognition of microscopic images of traditional Chinese medicinal materials.