A lightweight and efficient two-stage video flame detection algorithm was designed to address issues of high false positive rates, poor adaptability, and low efficiency in complex scenes. In the first stage, an improved adaptive Gaussian mixture model(AGMM) was employed for rapid background modeling of video image sequences. Suspicious candidate regions were extracted from the sequences by leveraging the flickering and surging characteristics of flames. In the second stage, a residual deep normalization and convolutional neural network (ResDN) was used to discriminate these suspicious candidate regions. A simplified residual block was introduced to replace the original convolutional layers for a lightweight design, enabling accurate flame detection and localization. Compared with traditional classification algorithms, the proposed two-stage video flame detection algorithm effectively overcomes environmental interference in complex scenes, rapidly and accurately identifies flames, and demonstrates higher detection rates and adaptability.