To address the issues of low interaction efficiency and non-intuitive control methods of traditional robotic arms in complex scenarios, a robotic arm interaction control system based on multi-dimensional gesture recognition under visual perception was designed. Three types of interaction control functions were accomplished, including static gesture interaction for the two-dimensional joint posture of the hand, dynamic gesture interaction for the three-dimensional movement of the hand’s end, and following control interaction for the hand’s movement trajectory. In this way, intuitive and efficient robot manipulation was enabled.The static gesture joint interaction was realized by combining an improved YOLO (you only look once) with the MediaPipe framework to achieve precise joint control through the detection of hand key points, with an identification accuracy rate of 93.9%. The dynamic gesture end movement interaction was implemented using 3DResNet and Bi-LSTM(bidirectional long short-term memory) networks to extract the spatiotemporal features of dynamic gestures, achieving an identification accuracy rate of 93.87%, and was used to drive the robotic arm’s end to move according to predefined instructions. The hand movement trajectory following control interaction was achieved by capturing three-dimensional coordinates with a depth camera and combining Kalman filtering and sliding mode controllers to ensure smooth trajectory tracking, with an error controlled within 4.5 mm.Through simulation and physical experiments conducted on UR5 and Dobot robotic arms, the effectiveness and accuracy of these three interaction functions are verified, providing an efficient, natural, and cross-platform solution for human-robot interaction.