Aiming at several problems faced by the existing perimeter intrusion detection technology when applied to private houses, an integrated detection algorithm for perimeter intrusion for courtyard houses was proposed. The algorithm was based on YOLOv5, and the perimeter intrusion detection task was divided into two parts: abnormal object detection and intrusion detection. In the abnormal object detection part, the network structure of YOLOv5 was improved by three points in view of its poor detection effect on small objects in complex outdoor scenes. Then, the improved YOLOv5 was applied to detect whether the tools worn and held by personnel within the monitoring range are abnormal, and the potential intrusion behavior was determined according to the results. In the part of intrusion detection, a point-line area intrusion detection method was proposed, which cleverly abstracts the problem of area intrusion detection into the judgment of the position relationship between a point and the area enclosed by multiple lines. The experimental results show that the mean average precision (mAP) of the proposed algorithm in the abnormal object detection stage is 85.4%, which is higher than that of YOLOv5 and other object detection algorithms. The model checking speed can reach 23.4 frame/s, and the real-time performance is good. In the intrusion detection stage, compared with the existing video-based intrusion detection methods, the pointline area intrusion detection method has higher intrusion detection sensitivity and no false detection phenomenon. It basically meets the task requirements of perimeter intrusion detection for courtyard houses.