To clarify the hierarchical structure, interaction relationships, and influence pathways of the factors affecting outdoor thermal comfort, and to improve the theoretical framework of outdoor thermal comfort, the factors influencing outdoor thermal comfort were classified and systematically summarized. The interpretive structural modeling and matrices impacts croises-multiplication appliance classement (ISM-MICMAC) method was adopted. A hierarchical structural model of the factor system was constructed. Its driving-dependence characteristics were analyzed. The results show that the nine factors are divided into a four-level hierarchical structure. Climatic characteristics are the fundamental driving factor. Microclimate factors, street-scale spatial form, and social factors are the key driving factors at the third level. Thermal experience and individual factors are the second-level factors linking environmental stimuli and individual perception. Physiological, psychological, and behavioral factors are the surface-level direct factors influencing thermal comfort. The MICMAC analysis classifies the factors into three categories: independent factors, autonomous factors, and dependent factors. No linkage factors are identified. The functional positions of the factors within the system are thus clarified. The interaction pathways and system structure of the factors influencing outdoor thermal comfort are revealed. A framework is provided for research on the influencing mechanisms of outdoor thermal comfort. The theoretical framework of outdoor thermal comfort is also enriched and improved.