In the intelligent diagnosis of pancreatic lesions by ultrasonic endoscopy, a diagnostic model based on the twin network architecture was proposed to address the problems of misdetection, omission and low accuracy in the detection of pancreatic lesions in different age groups and genders, as well as the detection of pancreatic lesions in multiple sites. The model consists of two improved ConvNeXt models, which perform the task of pancreatic site classification and the task of locating cancerous areas without labelling the cancerous sites with Grad_CAM, respectively. The improved ConNeXt model uses feature fusion to combine detailed texture and abstract semantic information, and the fusion ratio relationship of feature vectors is solved through experimental exploration to improve the classification accuracy and stability of the model. Experiments show that the twin ConvNeXt network model improved by feature fusion achieves a classification accuracy of 98.34% for pancreatic sites and up to 99.47% for cancerous sites, and the activation regions generated by Grad_CAM are highly consistent with the real cancerous regions, which provides an effective auxiliary diagnosis for doctors.