The disorderly charging behavior of large-scale electric vehicles (EVs) is recognized to cause prolonged queuing times, increased charging costs, unbalanced utilization of charging station resources, and aggravated voltage fluctuations in the power grid, thereby undermining system stability. To address these challenges, a hierarchical graph reinforcement learning (HGRL) algorithm was proposed to optimize EV charging navigation strategies for coordinated multi-agent benefits. An EV charging model that accounts for user heterogeneity was developed by incorporating mileage anxiety, temporal and economic preferences, and energy consumption. Based on this model, a multi-agent objective function was formulated to minimize the total time cost, economic cost, mileage anxiety, and driving energy consumption of users. To effectively solve the problem, the charging navigation task was modeled as a two-layer stochastic optimization problem encompassing both charging station destination selection and route planning. The proposed HGRL method is designed to overcome the curse of dimensionality in high-dimensional state spaces and the challenges of stochastic uncertainty in optimization. It is demonstrated through simulation results that the HGRL algorithm effectively optimizes the interests of multiple stakeholders—including EV users, charging stations, and the power grid—while providing personalized and intelligent charging navigation strategies that account for user heterogeneity, and exhibits strong adaptability to varying system conditions.