BTK (Bruton's tyrosine kinase) is played a crucial role in various biological processes, including B-cell malignancies and autoimmune diseases. Thus, the inhibition of BTK represents an effective strategy for treating a broad range of diseases. The objective of this study is to develop a machine learning-based virtual screening method for the identification of potential BTK inhibitors. First, a dataset consisting of 3 499 active BTK inhibitors and 7 927 inactive compounds was collected. Then, 18 classification models and two integrated classification models were constructed using three molecular descriptors and six machine learning algorithms. The model performance evaluation was conducted, showing that the integrated model constructed based on ECFP (4extended-connectivity fingerprint with diameter 4) descriptors and DNN (deep neural network), RF (random forest), SVM (support vector machine) algorithms was more accurate and reliable Next, 93 potential novel BTK inhibitors were screened from 5 million molecules in the ZINC database using the best-integrated model. Their binding modes to BTK protein crystals were further analyzed by molecular docking. Ultimately, six BTK inhibitors exhibiting high affinity are identified based on their potential to form stable hydrogen bond interactions with critical amino acid residues, namely Thr474, Glu475, Met477, Cys481, and Asp539 within the enzyme's active pocket. The predicted ADMET (absorption, distribution, metabolism, excretion and toxicity) parameters suggest that the identified candidate compounds possess favorable pharmacokinetic parameters and safety profiles. Molecular dynamics simulation further demonstrates that these compounds can stably bind to BTK protein and are expected to be lead compounds for the development of novel BTK inhibitors.