Traditional data-driven fault diagnosis methods are hard to accurately diagnosis faults in the reactor coolant system (RCS) of a nuclear power plant. In order to address this challenge, a model for fault diagnosis of the RCS was established in the following route. Firstly, recursive feature elimination with cross-validation (RFECV) algorithm was used to select the features for the model. Then, improved Whale Optimization Algorithm (IWOA) was used to tune the hyperparameters of the model. Finally, the XGBoost model was used to establish the fault diagnosis model for RCS. The established model was applied to diagnose loss of coolant accident (LOCA), main pump trip (MPT), and steam generator tube rupture (SGTR), and the accuracy of the established model was validated by comparing with the traditional data-driven fault diagnosis models. The diagnosis result could be treated as a crucial reference for ensuring the safe and stable operation of the nuclear reactor and preventing nuclear accidents.