To address the cumbersome calibration process of fisheye cameras and its inapplicability to everyday scene images, a novel convolutional neural network(CNN)-based method was proposed that simultaneously calibrates the intrinsic parameters of fisheye lenses and corrects image distortion. The accuracy of fisheye camera calibration and image distortion correction was improved by predicting the displacement of pixel points under different distortion parameters. A coordinate attention module was introduced in the encoding part to enhance the model's accuracy and generalization ability to increase attention to image position information. Additionally, a cross-scale fusion module was designed in the skip connections to enhance image detail features. To address the issues of dataset scarcity and incomplete distortion parameter distribution, a new large-scale dataset labeled with corresponding distortion parameters and images after distortion correction was created. Experimental results show that compared to other fisheye camera calibration methods, this method achieves a reprojection error of 0.312 pixel, indicating the highest calibration accuracy. Additionally, compared to other image distortion correction methods, a peak signal to noise ratio(PSNR) of 38.055 dB and an structural similarity(SSIM) of 0.874 are achieved, indicating the best quality of image distortion correction.