A deep learning based T-beam formwork polishing robot was designed for the problems of difficult and time consuming polishing of T-beam formwork for variable cross-section. Firstly, an adaptive polishing structure was proposed to solve the technical problem that the existing polishing device cannot fit the inner variable cross-section of the T-beam formwork, and the polishing roller was easy to get stuck in the T-beam formwork partition. Secondly, in order to realize the quantitative monitoring of the polishing quality, a YOLOv8n-DSE algorithm was proposed to identify concrete dirt and stains on the formwork, the DySample dynamic up-sampling module was introduced to enhance the anti-interference ability of the model and accelerate the calculation speed, to improve the accuracy of small target detection, the SOEP (small object enhance pyramid) module was designed to improve the detection performance of small target detection through the SPDConv(space to depth convolution) to obtain the information features of the small target and give them to the CSP(cross stage partial)-Omni-Kernel for the integration of the features. Finally, the EMA(exponential moving average)-SlideLoss was replaced to make the model more concerned with the quantitative monitoring of the concrete, allowed the model to focus more on difficult targets, which can improve the effect on difficult case detection. The accuracy, recall, and mAP(mean average precision) values are improved by 3.1%, 9.7%, and 3.2%, respectively, compared with those before the improvement. The improved model was deployed to the robot and tested in the field. The results show that the equipment meets the plant's needs for polishing variable-section T-beam formwork.