Great progress has been made in the research of speaker recognition based on machine learning theory.This paper is based on the study of speaker recognition by KELM and MFCC to start a study on PSO and PCA optimized KELM speaker recognition algorithms,the MFCC was optimized by PCA and the initial input parameters of KELM was optimized by PSO.The improved algorithm was simulated on MATLAB platform and the performance comparative analysis is made by using the BP and SVM provided by MATLAB toolbox.The simulation results show that Optimized by PSO and PCA fusion to improve KELM, extreme learning machine can be arbitrary initial input parameters and does not require iteration update and it can effectively overcome the unstable performance due to the random determination of the initial weight, further improve the accuracy of classification matching and have a good application value.