With the rapid development of rare earth industry, site heavy metal pollution caused by smelting processes has been aggravated, which severely restricts the green transformation of this industry. Accurate identification of pollution characteristics, spatial distribution and sources of soil heavy metals in smelting sites is essential for green development promotion and contaminated site remediation of rare earth industry. In this study, 259 soil samples were collected from three layers with different depths at an ion-adsorption rare earth smelting site in South China. A three-dimensional spatial pollution field model was constructed, and geo-accumulation index, pollution load index and Nemerow comprehensive index were adopted to quantitatively evaluate the pollution degrees of seven heavy metals including As, Cd, Pb, Cu, Hg, Ni and Co. GIS technology, PMF model and APCS-MLR model were jointly used to analyze the spatial distribution and quantitatively apportion pollution sources of soil heavy metals. The results indicate that heavy metals in the study area follow a vertical distribution rule of surface input, local accumulation and deep attenuation. The studied soil is overall acidic. The contents of Cd and Hg in partial sampling points exceed the background values of Jiangxi red soil, and As, Ni and Co concentrations at some points are higher than soil risk screening values. The over-standard rate of Co in 0–0.5 m topsoil reaches 12.50%, with a maximum exceeding multiple of 3.33. Significant spatial differentiation of heavy metals is observed, and high-value zones are concentrated around waste residue warehouse, acid dissolution workshop and roasting workshop, while the overall pollution risk remains controllable. Hg, Cd, Co and Pb are identified as major pollutant elements by multi-index evaluation. The topsoil presents no obvious pollution generally, except moderate to severe pollution detected near the waste residue warehouse. As enrichment is mainly controlled by geological background. Three pollution sources are classified by two receptor models: parent material source (36.4% and 30.4%), smelting waste residue source (53.0% and 50.4%) and industrial kiln source (10.6% and 19.2%), which dominate the distribution of As, Co and Hg respectively. The integrated method combining GIS with PMF and APCS-MLR can provide scientific support for precise pollution control and differentiated remediation of rare earth smelting sites.