The common methods for displacement measurement of bridge structures using computer vision include feature point algorithm and optical flow method. To address the influence of illumination variation on the accuracy of the optical flow method and the difficulty of the feature point method in real-time tracking of the target, the target feature points were extracted using the oriented FAST and rotated BRIEF (ORB) feature point algorithm, and tracking was carried out in combination with the Lucas-Kanade(LK) optical flow method. By complementing and integrating the two algorithms and setting a periodic update strategy, the accumulation of errors can be effectively reduced. The vibration test of the steel plate beam model was conducted indoors to explore the influence of light variation on the performance of the algorithm. The displacement data obtained by the algorithm is compared with the measurement results of the laser displacement meter to verify its accuracy. In addition, the algorithm was compared and analyzed with algorithms such as FAST-LK, SIFT-LK, Shi-Tomasi-LK, and siamRPN + +. The results show that the displacement curve fitting degree obtained by this algorithm is above 95%, and the error is controlled within 5%, verifying its feasibility. Meanwhile, the experiments indicate that this method has certain robustness to illumination changes and can meet the requirements of real-time performance.