Complex channel networks are characterized by long navigation distances, multiple entry/exit points, diverse navigation segments and routes, and varying navigation classification levels. These features substantially increase the difficulty of vessel entry and departure scheduling in ports. A simulation-optimization framework was proposed to jointly optimize route selection and sequence scheduling decisions for inbound and outbound vessels. Specifically, the discrete-event and agent-based simulation model simulated vessel movements in complex channel networks under multiple navigation constraints, including one-/two-way traffic regulations, tidal windows, safety separation requirements, and night-navigation restrictions. The simulation dynamically resolved traffic conflicts and interactive vessel behaviors (e.g., head-on encounters, collision avoidance, and waiting), and output time stamps of key operational stages for each vessel, thereby providing a reliable basis for schedule evaluation and subsequent optimization. In the optimization module, an improved genetic algorithm (IGA) incorporating the Metropolis acceptance criterion from simulated annealing was employed. The objective was to minimize the total waiting time of all vessels in the port. By iteratively interacting with the simulation model, a closed-loop “simulation-optimization” procedure was formed to continuously refine scheduling solutions. Computational experiments demonstrate that the proposed simulation model can effectively reproduce the full vessel entry-departure process in complex channel networks and rigorously accommodate complicated navigation constraints. Compared with conventional genetic algorithms and simulated annealing, the IGA achieves significantly better convergence speed and solution quality, with more pronounced advantages in high-traffic scenarios where scheduling conflicts occur frequently. Overall, the proposed approach provides a robust theoretical basis to support scientific decision-making for vessel scheduling in ports with complex channel networks.