Rainfall infiltration is one of the main factors inducing slope instability. Based on the modeling method coupled with finite volume method and discrete element method, a numerical calculation model of gravelly granite residual soil slope under rainfall condition was established, and the reliability of the numerical model was verified by comparing with the indoor slope model test. The numerical calculation model was used to analyze the influence of different gravel contents on the stability of gravelly granite residual soil slope under rainfall infiltration, and to study the change rules of the particle displacement field, rotation field, contact force chain, constitutive anisotropy and so on. The effect of different gravel content on the stability of gravelly granite residual slope under rainfall infiltration was analyzed by numerical modeling, and the variation rules of particle displacement field, rotation field, contact force chain, and structural anisotropy were studied. The results show as follows. Under the action of rainfall infiltration, the sliding surface is circular sliding when the gravel content is 0 and 25%, and it is flowing sliding when it is 50%. With the increase of gravel content, the contact long force chain gradually decreases, the contact short force chain gradually increases, and the continuity of contact force decreases, and the contact type of the main contact that transmits the stresses is changed from soil-soil contact to soil-rock and stone-rock contact type. The contact anisotropy of slope particles shows the same pattern during the rainfall process, and the degree of contact anisotropy is also changed in the rainfall process. The degree of contact anisotropy between slope particles changes dynamically during rainfall and increases significantly when the gravel content increases to 75%. The stability of the slope decreases and then increases with the gravel content, and the stability of the slope increases significantly when the gravel content reaches 75%.