Coal pile is one of the common faults of conveyors. In order to ensure the safety of coal mine industrial production，it is necessary to detect the coal pile situation of underground conveyors in coal mines. However，the existing detection methods have the disadvantages of easy false touch and poor detection reliability. To solve these problems，a unified multi-scale temporal ConvTransformer （UnMS-TCT） network based on Transformer was proposed for conveyor coal pile detection. Firstly，the features extracted from RGB frames and optical flow frames were fused to enable the network to model the spatiotemporal relationship more comprehensively. Then，in the temporal encoder，the global module composed of dynamic position embedding （DPE），multi-head relation aggregator （MHRA） and multilayer perceptron （MLP） and the local module composed of cross-attention （CA） were alternately formed into a global-local relationship module to enhance the ability to obtain global and local temporal relationships at multiple scales. Secondly，the residual global-local fusion （ResGLFus） module was used to fuse multi-scale features，effectively improving the stability of the fusion process，and finally achieving high-precision coal pile prediction. The experimental results show that this method can detect coal piles on conveyors，and the mAP reaches 98.17%.