The Rotary Squeeze Cast-in-Place pile (RSCP) is the pile with diverse cross-sections at different depth, and it is formed by extruding the hole at different depth. To explore the bearing mechanism of RSCP, the centrifuge tests of the RSCP with two or three enlargements were conducted. The change law of the enlargement head end resistance and the pile tip resistance of RSCPs under different upper load were explored. It was found that at the later stage of loading , the stable percentage relationship appears among the enlargement head end resistance, the pile tip resistance, and the upper load when considering the RSCP with more than one enlargement. The mathematical model of the settlement of the RSCP was established using area integration based on the Mindlin Solution. The stable percentage relationship among the enlargement head end resistance, the pile tip resistance, and the upper load was adopted to replace the enlargement head end resistance and the pile tip resistance using the upper load. Then, the settlement of RSCP can be calculated once the upper load, the size of the enlargement, and the characteristics of soil around the RSCP are determined. Finally, the calculation results was verified by the centrifuge test results, which showed that the mathematical model can predict the settlement and the ultimate bearing capacity of RSCP precisely.