In order to solve the problems of wind power fluctuations and intermittency during grid integration, which affect the stable operation of the power grid, a capacity optimization configuration scheme for a flywheel-lithium battery hybrid energy storage system was proposed. This scheme combined empirical mode decomposition (EMD) and variational mode decomposition (VMD). Firstly, typical daily data was obtained using the K-means algorithm, and EMD was applied to decompose the output power signal of these typical wind power daily data into grid-connected power that meets fluctuation limits and power that needs to be smoothed by the hybrid energy storage system. Then, the sparrow search algorithm was used to optimize the number of decomposition modes K and the quadratic penalty factor α in the VMD algorithm. By decomposing the power that needs smoothing using VMD, a reasonable allocation between lithium batteries and flywheel energy storage was achieved. Finally, considering the constraints of energy storage charging and discharging power and state of charge, an economic model was constructed with energy storage cost as the objective function. The actual power generation data of Qiejidunqu wind farm in Gonghe County, Hainan Prefecture, Qinghai Province were simulated and calculated by MATLAB platform. The results show that the proposed strategy not only effectively mitigates wind power fluctuations but also improves the overall economy of the system.