In rock image recognition, achieving rapid and accurate identification of rocks is crucial for the digitalization of rocks. Among the challenges faced in intelligent rock recognition is the issue of image blurring caused by environmental factors such as lighting and humidity. In light of this, a novel deep learning approach (MobileNetV3-small-RegNetX) was proposed for rock image recognition, which is suitable for scenarios with limited resources such as mobile devices. Building upon the RegNet network, transfer learning methods, combining the advantages of the MobileNetV3 residual structure with squeeze-and-excitation (SE) modules was employed to effectively optimize feature extraction and network structure, leading to a significant improvement in detection speed. To validate the accuracy of this approach, comparative experiments were conducted between the new model and current mainstream lightweight models (DenseNet and ShuffleNet). The results demonstrate that the new model proposed exhibits high precision (82.15%) and fast processing (0.06 GFLOPs). Additionally, the model demonstrates good adaptability to environmental factors such as lighting and humidity-induced image blurring.